You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flink.apache.org by "Tzu-Li (Gordon) Tai" <tz...@apache.org> on 2020/10/23 15:01:28 UTC

[DISCUSS] Releasing Apache Flink 1.11.3

Hi,

Xintong and I would like to start a discussion for releasing Flink 1.11.3
soon.

It seems like we already have a few pressing issues that needs to be
included in a new hotfix release:

   - Heap-based timers’ restore behaviour is causing a critical recovery
   issue for StateFun [1] [2] [3].
   - There are several robustness issues for the FLIP-27 new source API,
   such as [4]. We already have some users using the FLIP-27 API with 1.11.x,
   so it would be important to get those fixes in for 1.11.x as well.

Apart from the issues that are already marked as blocker for 1.11.3 in our
JIRA [5], please let us know in this thread if there is already ongoing
work for other important fixes that we should try to include.

Xintong and I would like to volunteer for managing this release, and will
try to communicate the priority of pending blockers over the next few days.
Since the aforementioned issues are quite critical, we’d like to aim
for a *feature
freeze by the end of next week (Oct. 30th)* and start the release voting
process the week after.
If that is too short of a notice and you might need more time, please let
us know!

Cheers,
Gordon

[1] https://issues.apache.org/jira/browse/FLINK-19692
[2] https://issues.apache.org/jira/browse/FLINK-19741
[3] https://issues.apache.org/jira/browse/FLINK-19748
[4] https://issues.apache.org/jira/browse/FLINK-19717
[5]
https://issues.apache.org/jira/issues/?jql=project%20%3D%20FLINK%20AND%20priority%20%3D%20Blocker%20AND%20fixVersion%20%3D%201.11.3

Re: [DISCUSS] Releasing Apache Flink 1.11.3

Posted by Till Rohrmann <tr...@apache.org>.
Thanks a lot for driving this release Xintong! Looking forward to the
1.11.3 release :-)

Cheers,
Till

On Tue, Dec 8, 2020 at 11:27 AM Xintong Song <to...@gmail.com> wrote:

> Hi devs,
>
> There has been a bit more delay but we finally get all bockers resolved.
>
> I'm creating the RC right now (commit id:
> 47c2d97499ed95a33c071e916eb8bde7b179e879).
> This could take some time given that I'm doing this for the first time and
> accessing to Apache Nexus Repository from my place is slow.
>
> Once the RC is created, I'll start a new voting thread.
>
> Thank you~
>
> Xintong Song
>
>
>
> On Fri, Dec 4, 2020 at 11:46 AM Xintong Song <to...@gmail.com>
> wrote:
>
> > Updates on the release progress.
> >
> > - FLIP-27 is stabilizing on the master branch. Becket is working on the
> > final backportings, which is estimated to be finished this weekend.
> > - There's a new blocker FLINK-20442, a license problem discovered during
> > the 1.12.0 RC2 testings, which is already fixed for 1.12.
> >
> > We are hoping to create a RC next Monday, if things go well for the above
> > issues.
> >
> >
> > Thank you~
> >
> > Xintong Song
> >
> >
> >
> > On Fri, Nov 27, 2020 at 9:49 AM Xintong Song <to...@gmail.com>
> > wrote:
> >
> >> Hi devs,
> >>
> >> Updates on the release progress.
> >>
> >> There are a couple of new blockers emerged since the last update, all of
> >> which have been resolved in the past week.
> >>
> >> The release 1.11.3 is still blocked on FLIP-27.
> >> Stephan and Becket are trying to first get the feature stabilized on the
> >> master branch for release 1.12. After that, the remaining changes will
> be
> >> backported to 1.11, and the release 1.11.3 will be unblocked.
> >>
> >> We are hoping to deliver the first RC by the end of next week. During
> >> this time, if there's any other things that you believe should be
> included
> >> in this release, please reach out to either me or Gordon.
> >>
> >> Sorry for the delay.
> >>
> >> Thank you~
> >>
> >> Xintong Song
> >>
> >>
> >>
> >> On Mon, Nov 23, 2020 at 2:03 PM Xintong Song <to...@gmail.com>
> >> wrote:
> >>
> >>> Thanks for the notice, Zhu.
> >>> Let's have the discussion on the jira ticket.
> >>>
> >>> Thank you~
> >>>
> >>> Xintong Song
> >>>
> >>>
> >>>
> >>> On Mon, Nov 23, 2020 at 12:10 PM Zhu Zhu <re...@gmail.com> wrote:
> >>>
> >>>> Hi Xintong,
> >>>> We just found an issue FLINK-20285 and I think it is a blocker for
> >>>> 1.11.3.
> >>>>
> >>>> Thanks,
> >>>> Zhu
> >>>>
> >>>> Gyula Fóra <gy...@gmail.com> 于2020年11月18日周三 下午9:36写道：
> >>>>
> >>>> > Hi All!
> >>>> >
> >>>> > I have found the following issue today which might be considered a
> >>>> blocker
> >>>> > for this release as well:
> >>>> >
> >>>> > https://issues.apache.org/jira/browse/FLINK-20221
> >>>> >
> >>>> > Could someone please quickly provide a second set of eyes and
> confirm
> >>>> that
> >>>> > this is indeed a big problem? :)
> >>>> >
> >>>> > Thank you!
> >>>> > Gyula
> >>>> >
> >>>> > On Wed, Nov 18, 2020 at 5:12 AM Xintong Song <tonysong820@gmail.com
> >
> >>>> > wrote:
> >>>> >
> >>>> > > Hi devs,
> >>>> > >
> >>>> > > Updates on the progress of preparing the 1.11.3 release.
> >>>> > >
> >>>> > > We are approaching the creation of our first release candidate.
> >>>> Thanks
> >>>> > all
> >>>> > > for the efforts so far.
> >>>> > >
> >>>> > > # Ongoing threads
> >>>> > >
> >>>> > >    - *FLIP-27 backportings:* 74 out of 102 commits are already
> >>>> > backported.
> >>>> > >    Stephan and Becket are actively working on this.
> >>>> > >    - *Blockers:* There are 2 remaining blockers, both have been
> >>>> fixed on
> >>>> > >    the master branch and should be backported along with the
> FLIP-27
> >>>> > > changes.
> >>>> > >
> >>>> > > As soon as the above threads are done, we will create our first
> >>>> release
> >>>> > > candidate. If there's any other issues that you believe should be
> a
> >>>> > release
> >>>> > > blocker, please reach out to either me or Gordon.
> >>>> > >
> >>>> > > # JIRA version
> >>>> > >
> >>>> > > Version 1.11.4 has been created on JIRA. I've already tagged some
> >>>> of the
> >>>> > > tickets with the new fix version. Please feel free to continue
> with
> >>>> the
> >>>> > > efforts and merge once they are ready. I'll double check and
> update
> >>>> the
> >>>> > fix
> >>>> > > versions before the release.
> >>>> > >
> >>>> > > Thank you~
> >>>> > >
> >>>> > > Xintong Song
> >>>> > >
> >>>> > >
> >>>> > >
> >>>> > > On Thu, Nov 12, 2020 at 1:31 PM Xintong Song <
> tonysong820@gmail.com
> >>>> >
> >>>> > > wrote:
> >>>> > >
> >>>> > > > Thanks for the notice and fix, Roman.
> >>>> > > >
> >>>> > > > Thank you~
> >>>> > > >
> >>>> > > > Xintong Song
> >>>> > > >
> >>>> > > >
> >>>> > > >
> >>>> > > > On Wed, Nov 11, 2020 at 5:53 PM Khachatryan Roman <
> >>>> > > > khachatryan.roman@gmail.com> wrote:
> >>>> > > >
> >>>> > > >> Hi,
> >>>> > > >>
> >>>> > > >> I'd like FLINK-20079 [1] to be merged into 1.11 and included in
> >>>> > 1.11.3.
> >>>> > > >>
> >>>> > > >> [1] https://issues.apache.org/jira/browse/FLINK-20079
> >>>> > > >>
> >>>> > > >> Regards,
> >>>> > > >> Roman
> >>>> > > >>
> >>>> > > >>
> >>>> > > >> On Tue, Nov 10, 2020 at 8:21 AM Xintong Song <
> >>>> tonysong820@gmail.com>
> >>>> > > >> wrote:
> >>>> > > >>
> >>>> > > >> > Thanks for the notice, Dian.
> >>>> > > >> >
> >>>> > > >> > Thank you~
> >>>> > > >> >
> >>>> > > >> > Xintong Song
> >>>> > > >> >
> >>>> > > >> >
> >>>> > > >> >
> >>>> > > >> > On Tue, Nov 10, 2020 at 10:19 AM Dian Fu <
> >>>> dian0511.fu@gmail.com>
> >>>> > > wrote:
> >>>> > > >> >
> >>>> > > >> > > Hi Xintong,
> >>>> > > >> > >
> >>>> > > >> > > I want to bring one more issue to your attention [1]. The
> >>>> test
> >>>> > case
> >>>> > > >> > > UnalignedCheckpointCompatibilityITCase.test failed several
> >>>> times
> >>>> > in
> >>>> > > >> the
> >>>> > > >> > > last nightly test of release-1.11. We need to figure out if
> >>>> it's
> >>>> > > just
> >>>> > > >> an
> >>>> > > >> > > instable test or caused by recent changes.
> >>>> > > >> > >
> >>>> > > >> > > [1] https://issues.apache.org/jira/browse/FLINK-20065
> >>>> > > >> > >
> >>>> > > >> > > > 在 2020年11月10日，上午9:24，Xintong Song <tonysong820@gmail.com
> >
> >>>> 写道：
> >>>> > > >> > > >
> >>>> > > >> > > > Thanks for the replies.
> >>>> > > >> > > >
> >>>> > > >> > > > Thank you~
> >>>> > > >> > > >
> >>>> > > >> > > > Xintong Song
> >>>> > > >> > > >
> >>>> > > >> > > >
> >>>> > > >> > > >
> >>>> > > >> > > > On Tue, Nov 10, 2020 at 1:09 AM Becket Qin <
> >>>> > becket.qin@gmail.com>
> >>>> > > >> > wrote:
> >>>> > > >> > > >
> >>>> > > >> > > >> Hi Xintong,
> >>>> > > >> > > >>
> >>>> > > >> > > >> Thanks for driving the release. Just want to sync up on
> >>>> the
> >>>> > > FLIP-27
> >>>> > > >> > > >> backporting. Stephan and I are still trying to backport
> a
> >>>> bunch
> >>>> > > of
> >>>> > > >> > > patches
> >>>> > > >> > > >> of Source to 1.11.3. Including:
> >>>> > > >> > > >>
> >>>> > > >> > > >> [FLINK-19698][connector/common] Add a close() method to
> >>>> the
> >>>> > > >> > SplitReader.
> >>>> > > >> > > >> [FLINK-19717] SourceReaderBase.pollNext may return
> >>>> END_OF_INPUT
> >>>> > > if
> >>>> > > >> > > >> SplitReader.fetch throws
> >>>> > > >> > > >> [FLINK-19535] [connector/common] Avoid failing a job
> >>>> multiple
> >>>> > > >> times in
> >>>> > > >> > > >> SourceCoordinator.
> >>>> > > >> > > >> [FLINK-19265] [FLINK-20049][core] Source API final
> >>>> adjustments.
> >>>> > > >> > > >>
> >>>> > > >> > > >> and a few more fixes.
> >>>> > > >> > > >>
> >>>> > > >> > > >> We are currently trying to fix them in 1.12 first so it
> >>>> might
> >>>> > > take
> >>>> > > >> a
> >>>> > > >> > > little
> >>>> > > >> > > >> longer to backport them to 1.11.3. I think it will
> >>>> probably
> >>>> > take
> >>>> > > >> us a
> >>>> > > >> > > few
> >>>> > > >> > > >> more days to finish the backport. So that would roughly
> >>>> be the
> >>>> > > end
> >>>> > > >> of
> >>>> > > >> > > this
> >>>> > > >> > > >> week.
> >>>> > > >> > > >>
> >>>> > > >> > > >> Thanks,
> >>>> > > >> > > >>
> >>>> > > >> > > >> Jiangjie (Becket) Qin
> >>>> > > >> > > >>
> >>>> > > >> > > >>
> >>>> > > >> > > >>
> >>>> > > >> > > >>
> >>>> > > >> > > >> On Mon, Nov 9, 2020 at 9:57 PM Till Rohrmann <
> >>>> > > trohrmann@apache.org
> >>>> > > >> >
> >>>> > > >> > > wrote:
> >>>> > > >> > > >>
> >>>> > > >> > > >>> Yes, I've downgraded FLINK-19816 to critical.
> >>>> > > >> > > >>>
> >>>> > > >> > > >>> Cheers,
> >>>> > > >> > > >>> Till
> >>>> > > >> > > >>>
> >>>> > > >> > > >>> On Mon, Nov 9, 2020 at 10:19 AM Xintong Song <
> >>>> > > >> tonysong820@gmail.com>
> >>>> > > >> > > >>> wrote:
> >>>> > > >> > > >>>
> >>>> > > >> > > >>>> Thanks for the notice, Till.
> >>>> > > >> > > >>>>
> >>>> > > >> > > >>>> I just checked and found FLINK-20033 is already fixed.
> >>>> Shall
> >>>> > we
> >>>> > > >> also
> >>>> > > >> > > >>>> downgrade FLINK-19816 to `Critical`?
> >>>> > > >> > > >>>>
> >>>> > > >> > > >>>> Thank you~
> >>>> > > >> > > >>>>
> >>>> > > >> > > >>>> Xintong Song
> >>>> > > >> > > >>>>
> >>>> > > >> > > >>>>
> >>>> > > >> > > >>>>
> >>>> > > >> > > >>>> On Mon, Nov 9, 2020 at 4:42 PM Till Rohrmann <
> >>>> > > >> trohrmann@apache.org>
> >>>> > > >> > > >>> wrote:
> >>>> > > >> > > >>>>
> >>>> > > >> > > >>>>> I would like to bring one more critical issue to your
> >>>> > > attention
> >>>> > > >> > which
> >>>> > > >> > > >>> is
> >>>> > > >> > > >>>>> FLINK-20033 [1]. I believe that this issue is
> actually
> >>>> > causing
> >>>> > > >> what
> >>>> > > >> > > >> has
> >>>> > > >> > > >>>>> been reported in FLINK-19816 [2]. I hope to have it
> >>>> fixed by
> >>>> > > the
> >>>> > > >> > end
> >>>> > > >> > > >> of
> >>>> > > >> > > >>>>> today. Once FLINK-20033 is fixed, I think that we
> >>>> don't have
> >>>> > > to
> >>>> > > >> > block
> >>>> > > >> > > >>> the
> >>>> > > >> > > >>>>> release on FLINK-19816.
> >>>> > > >> > > >>>>>
> >>>> > > >> > > >>>>> [1]
> https://issues.apache.org/jira/browse/FLINK-20033
> >>>> > > >> > > >>>>> [2]
> https://issues.apache.org/jira/browse/FLINK-19816
> >>>> > > >> > > >>>>>
> >>>> > > >> > > >>>>> Cheers,
> >>>> > > >> > > >>>>> Till
> >>>> > > >> > > >>>>>
> >>>> > > >> > > >>>>> On Mon, Nov 9, 2020 at 4:05 AM Xintong Song <
> >>>> > > >> tonysong820@gmail.com
> >>>> > > >> > >
> >>>> > > >> > > >>>> wrote:
> >>>> > > >> > > >>>>>
> >>>> > > >> > > >>>>>> Hi devs,
> >>>> > > >> > > >>>>>>
> >>>> > > >> > > >>>>>> I'd like to provide an update on the progress of
> >>>> preparing
> >>>> > > >> release
> >>>> > > >> > > >>>>> 1.11.3.
> >>>> > > >> > > >>>>>>
> >>>> > > >> > > >>>>>> *Blockers*
> >>>> > > >> > > >>>>>> We currently have 3 remaining blockers. (3 resolved
> >>>> and 1
> >>>> > > >> emerged
> >>>> > > >> > > >>>>> compared
> >>>> > > >> > > >>>>>> to last week)
> >>>> > > >> > > >>>>>>
> >>>> > > >> > > >>>>>> - [FLINK-19698] Add close() method and
> >>>> > onCheckpointComplete()
> >>>> > > >> to
> >>>> > > >> > > >> the
> >>>> > > >> > > >>>>>> Source.
> >>>> > > >> > > >>>>>> The issue has been fixed on the master branch. It's
> >>>> > currently
> >>>> > > >> > > >> blocked
> >>>> > > >> > > >>>> on
> >>>> > > >> > > >>>>>> the FLIP-27 backportings to backport it to the 1.11
> >>>> branch.
> >>>> > > >> > > >>>>>>
> >>>> > > >> > > >>>>>> - [FLINK-19717] SourceReaderBase.pollNext may return
> >>>> > > >> END_OF_INPUT
> >>>> > > >> > > >> if
> >>>> > > >> > > >>>>>> SplitReader.fetch throws
> >>>> > > >> > > >>>>>> A PR has been opened and reviewed. From the
> >>>> discussions on
> >>>> > > the
> >>>> > > >> PR,
> >>>> > > >> > > >> it
> >>>> > > >> > > >>>>> looks
> >>>> > > >> > > >>>>>> close to mergeable.
> >>>> > > >> > > >>>>>>
> >>>> > > >> > > >>>>>> - [FLINK-19816] Flink restored from a wrong
> >>>> checkpoint (a
> >>>> > > very
> >>>> > > >> old
> >>>> > > >> > > >>> one
> >>>> > > >> > > >>>>> and
> >>>> > > >> > > >>>>>> not the last completed one)
> >>>> > > >> > > >>>>>> This is a newly emerged blocker and Matthias is
> >>>> working on
> >>>> > > it.
> >>>> > > >> > > >>>>>>
> >>>> > > >> > > >>>>>> *Test Instabilities*
> >>>> > > >> > > >>>>>> We currently have 27 test instabilities[1].
> >>>> > > >> > > >>>>>> AFAIK, none of them are as serious as to block the
> >>>> 1.11.3
> >>>> > > >> release.
> >>>> > > >> > > >>>>>>
> >>>> > > >> > > >>>>>> *FLIP-27 Backprotings*
> >>>> > > >> > > >>>>>>
> >>>> > > >> > > >>>>>> I noticed that there's no jira issues opened on the
> >>>> FLIP-27
> >>>> > > >> > > >>> backporting
> >>>> > > >> > > >>>>>> efforts, which is part of the major efforts planned
> >>>> for the
> >>>> > > >> 1.11.3
> >>>> > > >> > > >>>>> release,
> >>>> > > >> > > >>>>>> making it hard to track the progress.
> >>>> > > >> > > >>>>>>
> >>>> > > >> > > >>>>>>
> >>>> > > >> > > >>>>>> @Stephan and @Becket, could you please share the
> >>>> updates on
> >>>> > > the
> >>>> > > >> > > >>>>> backporting
> >>>> > > >> > > >>>>>> efforts? How is the progress and when are the
> efforts
> >>>> > > expected
> >>>> > > >> to
> >>>> > > >> > > >> be
> >>>> > > >> > > >>>>>> finished? It would be appreciated and helpful if we
> >>>> can
> >>>> > have
> >>>> > > a
> >>>> > > >> > jira
> >>>> > > >> > > >>>>> ticket
> >>>> > > >> > > >>>>>> to track the backporing progress publicly.
> >>>> > > >> > > >>>>>>
> >>>> > > >> > > >>>>>>
> >>>> > > >> > > >>>>>> Thank you~
> >>>> > > >> > > >>>>>>
> >>>> > > >> > > >>>>>> Xintong Song
> >>>> > > >> > > >>>>>>
> >>>> > > >> > > >>>>>>
> >>>> > > >> > > >>>>>> [1]
> >>>> > > >> > > >>>>>>
> >>>> > > >> > > >>>>>>
> >>>> > > >> > > >>>>>
> >>>> > > >> > > >>>>
> >>>> > > >> > > >>>
> >>>> > > >> > > >>
> >>>> > > >> > >
> >>>> > > >> >
> >>>> > > >>
> >>>> > >
> >>>> >
> >>>>
> https://issues.apache.org/jira/browse/FLINK-16908?filter=12348580&jql=project%20%3D%20FLINK%20AND%20resolution%20%3D%20Unresolved%20AND%20affectedVersion%20in%20(1.11.0%2C%201.11.1%2C%201.11.2%2C%201.11.3)%20AND%20labels%20%3D%20test-stability%20ORDER%20BY%20updated%20DESC%2C%20created%20DESC
> >>>> > > >> > > >>>>>>
> >>>> > > >> > > >>>>>> On Wed, Nov 4, 2020 at 3:58 PM Tzu-Li (Gordon) Tai <
> >>>> > > >> > > >>>> tzulitai@apache.org>
> >>>> > > >> > > >>>>>> wrote:
> >>>> > > >> > > >>>>>>
> >>>> > > >> > > >>>>>>> Thanks for the efforts so far with pushing for
> >>>> 1.11.3.
> >>>> > > >> > > >>>>>>>
> >>>> > > >> > > >>>>>>> I'd like to provide a mid-week update on how we're
> >>>> looking
> >>>> > > >> with
> >>>> > > >> > > >> all
> >>>> > > >> > > >>>> the
> >>>> > > >> > > >>>>>>> pending blockers and backports:
> >>>> > > >> > > >>>>>>>
> >>>> > > >> > > >>>>>>> *Blockers**:*
> >>>> > > >> > > >>>>>>>
> >>>> > > >> > > >>>>>>> Currently all blockers either have an open PR, or
> >>>> have
> >>>> > been
> >>>> > > >> > > >> merged.
> >>>> > > >> > > >>>>> I'll
> >>>> > > >> > > >>>>>>> highlight below the blockers *that still require
> >>>> reviewing
> >>>> > > >> > > >> efforts
> >>>> > > >> > > >>>> for
> >>>> > > >> > > >>>>>> them
> >>>> > > >> > > >>>>>>> to move forward:*
> >>>> > > >> > > >>>>>>>
> >>>> > > >> > > >>>>>>> - [FLINK-19909] Flink application in attach mode
> >>>> could not
> >>>> > > >> > > >>> terminate
> >>>> > > >> > > >>>>> when
> >>>> > > >> > > >>>>>>> the only job is canceled. PR:
> >>>> > > >> > > >>>>> https://github.com/apache/flink/pull/13911
> >>>> > > >> > > >>>>>>> - [FLINK-19717] SourceReaderBase.pollNext may
> return
> >>>> > > >> END_OF_INPUT
> >>>> > > >> > > >>> if
> >>>> > > >> > > >>>>>>> SplitReader.fetch throws. PR:
> >>>> > > >> > > >>>>> https://github.com/apache/flink/pull/13776
> >>>> > > >> > > >>>>>>>
> >>>> > > >> > > >>>>>>> The above PRs currently have no reviews at all yet.
> >>>> They
> >>>> > > seem
> >>>> > > >> to
> >>>> > > >> > > >>>>> already
> >>>> > > >> > > >>>>>>> have designated reviewers.
> >>>> > > >> > > >>>>>>>
> >>>> > > >> > > >>>>>>>
> >>>> > > >> > > >>>>>>> *Backports:*
> >>>> > > >> > > >>>>>>> - FLIP-27 robustness improvement backports: Stephan
> >>>> is
> >>>> > > >> currently
> >>>> > > >> > > >>>>> working
> >>>> > > >> > > >>>>>> on
> >>>> > > >> > > >>>>>>> backporting several FLIP-27 changes. There are no
> >>>> PRs yet
> >>>> > > for
> >>>> > > >> the
> >>>> > > >> > > >>>>>>> backported changes.
> >>>> > > >> > > >>>>>>> - DataStreamUtils.collect() refactorings backport:
> >>>> Steven
> >>>> > Wu
> >>>> > > >> > > >>>> mentioned
> >>>> > > >> > > >>>>> to
> >>>> > > >> > > >>>>>>> backport this, but AFAIK this isn't assigned to
> >>>> anyone
> >>>> > yet.
> >>>> > > >> > > >>>>>>>
> >>>> > > >> > > >>>>>>> *ETAs*:
> >>>> > > >> > > >>>>>>>
> >>>> > > >> > > >>>>>>> I'd like to request ETAs for the remaining
> >>>> backports, to
> >>>> > > >> prevent
> >>>> > > >> > > >> a
> >>>> > > >> > > >>>>> creep
> >>>> > > >> > > >>>>>> in
> >>>> > > >> > > >>>>>>> the scope of this bugfix release.
> >>>> > > >> > > >>>>>>>
> >>>> > > >> > > >>>>>>> We already have Flink users that would benefit from
> >>>> fixes
> >>>> > > that
> >>>> > > >> > > >> have
> >>>> > > >> > > >>>>> been
> >>>> > > >> > > >>>>>>> merged to release-1.11 already, so technically
> >>>> speaking
> >>>> > the
> >>>> > > >> > > >>> backports
> >>>> > > >> > > >>>>>>> should be considered "nice-to-have" (to the best of
> >>>> my
> >>>> > > >> knowledge
> >>>> > > >> > > >> of
> >>>> > > >> > > >>>> the
> >>>> > > >> > > >>>>>>> changes) and could potentially be moved to a
> >>>> follow-up
> >>>> > > 1.11.4.
> >>>> > > >> > > >>>>>>> Most notably, the Stateful Functions project is
> >>>> already
> >>>> > > >> waiting
> >>>> > > >> > > >> on
> >>>> > > >> > > >>>>> Flink
> >>>> > > >> > > >>>>>>> 1.11.3 to address critical recovery issues (please
> >>>> see the
> >>>> > > >> > > >> StateFun
> >>>> > > >> > > >>>>> 2.2.1
> >>>> > > >> > > >>>>>>> release discussion thread [1]).
> >>>> > > >> > > >>>>>>>
> >>>> > > >> > > >>>>>>> @Stephan Ewen <se...@apache.org> @Becket Qin <
> >>>> > > >> > > >> becket.qin@gmail.com
> >>>> > > >> > > >>>>
> >>>> > > >> > > >>>>>> could
> >>>> > > >> > > >>>>>>> you provide an ETA for the FLIP-27 backports? It
> >>>> would
> >>>> > help
> >>>> > > to
> >>>> > > >> > > >> get
> >>>> > > >> > > >>> a
> >>>> > > >> > > >>>>>> better
> >>>> > > >> > > >>>>>>> estimate to decide how we proceed here.
> >>>> > > >> > > >>>>>>>
> >>>> > > >> > > >>>>>>> Cheers,
> >>>> > > >> > > >>>>>>> Gordon
> >>>> > > >> > > >>>>>>>
> >>>> > > >> > > >>>>>>> [1]
> >>>> > > >> > > >>>>>>>
> >>>> > > >> > > >>>>>>>
> >>>> > > >> > > >>>>>>
> >>>> > > >> > > >>>>>
> >>>> > > >> > > >>>>
> >>>> > > >> > > >>>
> >>>> > > >> > > >>
> >>>> > > >> > >
> >>>> > > >> >
> >>>> > > >>
> >>>> > >
> >>>> >
> >>>>
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Releasing-StateFun-hotfix-version-2-2-1-td46239.html
> >>>> > > >> > > >>>>>>>
> >>>> > > >> > > >>>>>>> On Wed, Nov 4, 2020 at 3:16 PM Tzu-Li (Gordon) Tai
> <
> >>>> > > >> > > >>>>> tzulitai@apache.org>
> >>>> > > >> > > >>>>>>> wrote:
> >>>> > > >> > > >>>>>>>
> >>>> > > >> > > >>>>>>>>> The collect() utils can be picked back, I see no
> >>>> issue
> >>>> > > with
> >>>> > > >> > > >>> that
> >>>> > > >> > > >>>>> (it
> >>>> > > >> > > >>>>>> is
> >>>> > > >> > > >>>>>>>> isolated utilities).
> >>>> > > >> > > >>>>>>>>
> >>>> > > >> > > >>>>>>>> Just checking on all the requested backports
> >>>> mentioned in
> >>>> > > >> this
> >>>> > > >> > > >>>>> thread,
> >>>> > > >> > > >>>>>>> and
> >>>> > > >> > > >>>>>>>> figuring out which ones seem to still be
> unassigned
> >>>> /
> >>>> > open.
> >>>> > > >> > > >>>>>>>>
> >>>> > > >> > > >>>>>>>> Is someone working on backporting
> >>>> > > >> > > >>>>>>>>
> >>>> > > >> > > >>>>>>>
> >>>> > > >> > > >>>>>>
> >>>> > > >> > > >>>>>
> >>>> > > >> > > >>>>
> >>>> > > >> > > >>>
> >>>> > > >> > > >>
> >>>> > > >> > >
> >>>> > > >> >
> >>>> > > >>
> >>>> > >
> >>>> >
> >>>>
> https://github.com/apache/flink/commit/09a7a66b7313fea64817fe960a8da1265b428efc
> >>>> > > >> > > >>>>>>>> to release-1.11 at the moment?
> >>>> > > >> > > >>>>>>>>
> >>>> > > >> > > >>>>>>>> On Wed, Nov 4, 2020 at 1:39 AM Steven Wu <
> >>>> > > >> stevenz3wu@gmail.com
> >>>> > > >> > > >>>
> >>>> > > >> > > >>>>> wrote:
> >>>> > > >> > > >>>>>>>>
> >>>> > > >> > > >>>>>>>>> @Stephan Ewen <se...@apache.org> yeah, we can do
> >>>> that.
> >>>> > > >> don't
> >>>> > > >> > > >>>> worry
> >>>> > > >> > > >>>>>>> about
> >>>> > > >> > > >>>>>>>>> it. your earlier email had the perfect
> explanation
> >>>> on
> >>>> > why
> >>>> > > >> file
> >>>> > > >> > > >>>>> source
> >>>> > > >> > > >>>>>>>>> shouldn't be backported.
> >>>> > > >> > > >>>>>>>>>
> >>>> > > >> > > >>>>>>>>> On Tue, Nov 3, 2020 at 3:37 AM Stephan Ewen <
> >>>> > > >> sewen@apache.org
> >>>> > > >> > > >>>
> >>>> > > >> > > >>>>> wrote:
> >>>> > > >> > > >>>>>>>>>
> >>>> > > >> > > >>>>>>>>>> @Steven would it be possible to initially copy
> >>>> some of
> >>>> > > the
> >>>> > > >> > > >>> code
> >>>> > > >> > > >>>>> into
> >>>> > > >> > > >>>>>>> the
> >>>> > > >> > > >>>>>>>>>> iceberg source and later replace it by a
> >>>> dependency on
> >>>> > > the
> >>>> > > >> > > >>> Flink
> >>>> > > >> > > >>>>>> file
> >>>> > > >> > > >>>>>>>>>> source?
> >>>> > > >> > > >>>>>>>>>>
> >>>> > > >> > > >>>>>>>>>> On Mon, Nov 2, 2020 at 8:33 PM Steven Wu <
> >>>> > > >> > > >>> stevenz3wu@gmail.com>
> >>>> > > >> > > >>>>>>> wrote:
> >>>> > > >> > > >>>>>>>>>>
> >>>> > > >> > > >>>>>>>>>>> Stephan, thanks a lot for explaining the file
> >>>> > connector.
> >>>> > > >> > > >>> that
> >>>> > > >> > > >>>>>> makes
> >>>> > > >> > > >>>>>>>>>> sense.
> >>>> > > >> > > >>>>>>>>>>>
> >>>> > > >> > > >>>>>>>>>>> I was asking because we were trying to reuse
> >>>> some of
> >>>> > the
> >>>> > > >> > > >>>>>>>>> implementations
> >>>> > > >> > > >>>>>>>>>> in
> >>>> > > >> > > >>>>>>>>>>> the file source for Iceberg source. Flink
> Iceberg
> >>>> > source
> >>>> > > >> > > >>> lives
> >>>> > > >> > > >>>>> in
> >>>> > > >> > > >>>>>>> the
> >>>> > > >> > > >>>>>>>>>>> Iceberg repo, which is not possible to code
> >>>> against
> >>>> > the
> >>>> > > >> > > >>> master
> >>>> > > >> > > >>>>>>> branch
> >>>> > > >> > > >>>>>>>>> of
> >>>> > > >> > > >>>>>>>>>>> the Flink code.
> >>>> > > >> > > >>>>>>>>>>>
> >>>> > > >> > > >>>>>>>>>>> On Mon, Nov 2, 2020 at 3:31 AM Stephan Ewen <
> >>>> > > >> > > >>> sewen@apache.org
> >>>> > > >> > > >>>>>
> >>>> > > >> > > >>>>>>> wrote:
> >>>> > > >> > > >>>>>>>>>>>
> >>>> > > >> > > >>>>>>>>>>>> Hi Steven!
> >>>> > > >> > > >>>>>>>>>>>>
> >>>> > > >> > > >>>>>>>>>>>> So far there are no plans to pick back the
> file
> >>>> > system
> >>>> > > >> > > >>>>> connector
> >>>> > > >> > > >>>>>>>>> code.
> >>>> > > >> > > >>>>>>>>>>> This
> >>>> > > >> > > >>>>>>>>>>>> is still evolving and not finalized for 1.12,
> >>>> so I
> >>>> > > don't
> >>>> > > >> > > >>>> feel
> >>>> > > >> > > >>>>> it
> >>>> > > >> > > >>>>>>> is
> >>>> > > >> > > >>>>>>>>> a
> >>>> > > >> > > >>>>>>>>>>> good
> >>>> > > >> > > >>>>>>>>>>>> candidate to be backported.
> >>>> > > >> > > >>>>>>>>>>>> However, with the base connector changes
> >>>> backported,
> >>>> > > you
> >>>> > > >> > > >>>>> should
> >>>> > > >> > > >>>>>> be
> >>>> > > >> > > >>>>>>>>> able
> >>>> > > >> > > >>>>>>>>>>> to
> >>>> > > >> > > >>>>>>>>>>>> run the file connector code from master
> against
> >>>> > 1.11.3.
> >>>> > > >> > > >>>>>>>>>>>>
> >>>> > > >> > > >>>>>>>>>>>> The collect() utils can be picked back, I see
> no
> >>>> > issue
> >>>> > > >> > > >>> with
> >>>> > > >> > > >>>>> that
> >>>> > > >> > > >>>>>>>>> (it is
> >>>> > > >> > > >>>>>>>>>>>> isolated utilities).
> >>>> > > >> > > >>>>>>>>>>>>
> >>>> > > >> > > >>>>>>>>>>>> Best,
> >>>> > > >> > > >>>>>>>>>>>> Stephan
> >>>> > > >> > > >>>>>>>>>>>>
> >>>> > > >> > > >>>>>>>>>>>>
> >>>> > > >> > > >>>>>>>>>>>> On Mon, Nov 2, 2020 at 3:02 AM Steven Wu <
> >>>> > > >> > > >>>>> stevenz3wu@gmail.com>
> >>>> > > >> > > >>>>>>>>> wrote:
> >>>> > > >> > > >>>>>>>>>>>>
> >>>> > > >> > > >>>>>>>>>>>>> Basically, it would be great to get the
> latest
> >>>> code
> >>>> > in
> >>>> > > >> > > >>> the
> >>>> > > >> > > >>>>>>>>>>>>> flink-connector-files (FLIP-27).
> >>>> > > >> > > >>>>>>>>>>>>>
> >>>> > > >> > > >>>>>>>>>>>>> On Sat, Oct 31, 2020 at 9:57 AM Steven Wu <
> >>>> > > >> > > >>>>>> stevenz3wu@gmail.com
> >>>> > > >> > > >>>>>>>>
> >>>> > > >> > > >>>>>>>>>>> wrote:
> >>>> > > >> > > >>>>>>>>>>>>>
> >>>> > > >> > > >>>>>>>>>>>>>> Stephan, it will be great if we can also
> >>>> backport
> >>>> > > >> > > >> the
> >>>> > > >> > > >>>>>>>>>> DataStreamUtils
> >>>> > > >> > > >>>>>>>>>>>>>> related commits that help with collecting
> >>>> output
> >>>> > > >> > > >> from
> >>>> > > >> > > >>>>>>> unbounded
> >>>> > > >> > > >>>>>>>>>>>> streams.
> >>>> > > >> > > >>>>>>>>>>>>>> e.g.
> >>>> > > >> > > >>>>>>>>>>>>>>
> >>>> > > >> > > >>>>>>>>>>>>>>
> >>>> > > >> > > >>>>>>>>>>>>>>
> >>>> > > >> > > >>>>>>>>>>>>>
> >>>> > > >> > > >>>>>>>>>>>>
> >>>> > > >> > > >>>>>>>>>>>
> >>>> > > >> > > >>>>>>>>>>
> >>>> > > >> > > >>>>>>>>>
> >>>> > > >> > > >>>>>>>
> >>>> > > >> > > >>>>>>
> >>>> > > >> > > >>>>>
> >>>> > > >> > > >>>>
> >>>> > > >> > > >>>
> >>>> > > >> > > >>
> >>>> > > >> > >
> >>>> > > >> >
> >>>> > > >>
> >>>> > >
> >>>> >
> >>>>
> https://github.com/apache/flink/commit/09a7a66b7313fea64817fe960a8da1265b428efc
> >>>> > > >> > > >>>>>>>>>>>>>>
> >>>> > > >> > > >>>>>>>>>>>>>> I tried to copy and paste the code to
> unblock
> >>>> > > >> > > >> myself.
> >>>> > > >> > > >>>> but
> >>>> > > >> > > >>>>> it
> >>>> > > >> > > >>>>>>>>>> quickly
> >>>> > > >> > > >>>>>>>>>>>> got
> >>>> > > >> > > >>>>>>>>>>>>>> into the rabbit hole of more and more code.
> >>>> > > >> > > >>>>>>>>>>>>>>
> >>>> > > >> > > >>>>>>>>>>>>>> On Fri, Oct 30, 2020 at 11:02 AM Stephan
> Ewen
> >>>> <
> >>>> > > >> > > >>>>>>> sewen@apache.org
> >>>> > > >> > > >>>>>>>>>>
> >>>> > > >> > > >>>>>>>>>>>> wrote:
> >>>> > > >> > > >>>>>>>>>>>>>>
> >>>> > > >> > > >>>>>>>>>>>>>>> I have started with backporting the source
> >>>> API
> >>>> > > >> > > >>> changes.
> >>>> > > >> > > >>>>>> Some
> >>>> > > >> > > >>>>>>>>> minor
> >>>> > > >> > > >>>>>>>>>>>>>>> conflicts to solve, will need a bit more to
> >>>> finish
> >>>> > > >> > > >>>> this.
> >>>> > > >> > > >>>>>>>>>>>>>>>
> >>>> > > >> > > >>>>>>>>>>>>>>> On Fri, Oct 30, 2020 at 7:25 AM Tzu-Li
> >>>> (Gordon)
> >>>> > > >> > > >> Tai <
> >>>> > > >> > > >>>>>>>>>>>>> tzulitai@apache.org>
> >>>> > > >> > > >>>>>>>>>>>>>>> wrote:
> >>>> > > >> > > >>>>>>>>>>>>>>>
> >>>> > > >> > > >>>>>>>>>>>>>>>> @Stephan Ewen <se...@apache.org>
> >>>> > > >> > > >>>>>>>>>>>>>>>> Are there already plans or ongoing efforts
> >>>> for
> >>>> > > >> > > >>>>>> backporting
> >>>> > > >> > > >>>>>>>>> the
> >>>> > > >> > > >>>>>>>>>>> list
> >>>> > > >> > > >>>>>>>>>>>> of
> >>>> > > >> > > >>>>>>>>>>>>>>>> FLIP-27 changes that you posted?
> >>>> > > >> > > >>>>>>>>>>>>>>>>
> >>>> > > >> > > >>>>>>>>>>>>>>>> On Thu, Oct 29, 2020 at 7:08 PM Xintong
> >>>> Song <
> >>>> > > >> > > >>>>>>>>>>> tonysong820@gmail.com
> >>>> > > >> > > >>>>>>>>>>>>>
> >>>> > > >> > > >>>>>>>>>>>>>>>> wrote:
> >>>> > > >> > > >>>>>>>>>>>>>>>>
> >>>> > > >> > > >>>>>>>>>>>>>>>>> Hi folks,
> >>>> > > >> > > >>>>>>>>>>>>>>>>>
> >>>> > > >> > > >>>>>>>>>>>>>>>>> Just to provide some updates concerning
> the
> >>>> > > >> > > >> status
> >>>> > > >> > > >>>> on
> >>>> > > >> > > >>>>>> the
> >>>> > > >> > > >>>>>>>>>>>>>>>>> test instabilities.
> >>>> > > >> > > >>>>>>>>>>>>>>>>>
> >>>> > > >> > > >>>>>>>>>>>>>>>>> Currently, we have 30 unresolved tickets
> >>>> labeled
> >>>> > > >> > > >>>> with
> >>>> > > >> > > >>>>>>>>> `Affects
> >>>> > > >> > > >>>>>>>>>>>>> Version`
> >>>> > > >> > > >>>>>>>>>>>>>>>>> 1.11.x.
> >>>> > > >> > > >>>>>>>>>>>>>>>>>
> >>>> > > >> > > >>>>>>>>>>>>>>>>>
> >>>> > > >> > > >>>>>>>>>>>>>>>
> >>>> > > >> > > >>>>>>>>>>>>>
> >>>> > > >> > > >>>>>>>>>>>>
> >>>> > > >> > > >>>>>>>>>>>
> >>>> > > >> > > >>>>>>>>>>
> >>>> > > >> > > >>>>>>>>>
> >>>> > > >> > > >>>>>>>
> >>>> > > >> > > >>>>>>
> >>>> > > >> > > >>>>>
> >>>> > > >> > > >>>>
> >>>> > > >> > > >>>
> >>>> > > >> > > >>
> >>>> > > >> > >
> >>>> > > >> >
> >>>> > > >>
> >>>> > >
> >>>> >
> >>>>
> https://issues.apache.org/jira/browse/FLINK-19775?filter=12348580&jql=project%20%3D%20FLINK%20AND%20resolution%20%3D%20Unresolved%20AND%20affectedVersion%20in%20(1.11.0%2C%201.11.1%2C%201.11.2%2C%201.11.3)%20AND%20labels%20%3D%20test-stability%20ORDER%20BY%20created%20DESC
> >>>> > > >> > > >>>>>>>>>>>>>>>>>
> >>>> > > >> > > >>>>>>>>>>>>>>>>> Among the 30 tickets, 11 of them are:
> >>>> > > >> > > >>>>>>>>>>>>>>>>> - Have occured in the recent 3 months
> >>>> > > >> > > >>>>>>>>>>>>>>>>> - Not confirmed to be pure testability
> >>>> issues
> >>>> > > >> > > >>>>>>>>>>>>>>>>> - Not confirmed to be rare condition
> cases
> >>>> > > >> > > >>>>>>>>>>>>>>>>>
> >>>> > > >> > > >>>>>>>>>>>>>>>>> It would be nice if someone familiar with
> >>>> these
> >>>> > > >> > > >>>>>> components
> >>>> > > >> > > >>>>>>>>> can
> >>>> > > >> > > >>>>>>>>>>>> take a
> >>>> > > >> > > >>>>>>>>>>>>>>> look
> >>>> > > >> > > >>>>>>>>>>>>>>>>> into these issues.
> >>>> > > >> > > >>>>>>>>>>>>>>>>>
> >>>> > > >> > > >>>>>>>>>>>>>>>>> -
> >>>> > > >> > > >>> https://issues.apache.org/jira/browse/FLINK-17159
> >>>> > > >> > > >>>>>> (ES6)
> >>>> > > >> > > >>>>>>>>>>>>>>>>> -
> >>>> > > >> > > >>> https://issues.apache.org/jira/browse/FLINK-17912
> >>>> > > >> > > >>>>>>> (Kafka)
> >>>> > > >> > > >>>>>>>>>>>>>>>>> -
> >>>> > > >> > > >>> https://issues.apache.org/jira/browse/FLINK-17949
> >>>> > > >> > > >>>>>>> (Kafka)
> >>>> > > >> > > >>>>>>>>>>>>>>>>> ⁃
> >>>> > > >> > > >>> https://issues.apache.org/jira/browse/FLINK-18444
> >>>> > > >> > > >>>>>>> (Kafka)
> >>>> > > >> > > >>>>>>>>>>>>>>>>> -
> >>>> > > >> > > >>> https://issues.apache.org/jira/browse/FLINK-18634
> >>>> > > >> > > >>>>>>> (Kafka)
> >>>> > > >> > > >>>>>>>>>>>>>>>>> -
> >>>> > > >> > > >>> https://issues.apache.org/jira/browse/FLINK-18648
> >>>> > > >> > > >>>>>>> (Kafka)
> >>>> > > >> > > >>>>>>>>>>>>>>>>> -
> >>>> > > >> > > >>> https://issues.apache.org/jira/browse/FLINK-18807
> >>>> > > >> > > >>>>>>> (Kafka)
> >>>> > > >> > > >>>>>>>>>>>>>>>>> -
> >>>> > > >> > > >>> https://issues.apache.org/jira/browse/FLINK-19369
> >>>> > > >> > > >>>>>>>>>>>> (BlobClientTest)
> >>>> > > >> > > >>>>>>>>>>>>>>>>> -
> >>>> > > >> > > >>> https://issues.apache.org/jira/browse/FLINK-19436
> >>>> > > >> > > >>>>>>> (TPCDS)
> >>>> > > >> > > >>>>>>>>>>>>>>>>> -
> >>>> > > >> > > >>> https://issues.apache.org/jira/browse/FLINK-19690
> >>>> > > >> > > >>>>>>>>>>>> (Format/Parquet)
> >>>> > > >> > > >>>>>>>>>>>>>>>>> -
> >>>> > > >> > > >>> https://issues.apache.org/jira/browse/FLINK-19775
> >>>> > > >> > > >>>>>>>>>>>>>>>>> (SystemProcessingTimeServiceTest)
> >>>> > > >> > > >>>>>>>>>>>>>>>>>
> >>>> > > >> > > >>>>>>>>>>>>>>>>> Thank you~
> >>>> > > >> > > >>>>>>>>>>>>>>>>>
> >>>> > > >> > > >>>>>>>>>>>>>>>>> Xintong Song
> >>>> > > >> > > >>>>>>>>>>>>>>>>>
> >>>> > > >> > > >>>>>>>>>>>>>>>>>
> >>>> > > >> > > >>>>>>>>>>>>>>>>>
> >>>> > > >> > > >>>>>>>>>>>>>>>>> On Thu, Oct 29, 2020 at 10:21 AM Jingsong
> >>>> Li <
> >>>> > > >> > > >>>>>>>>>>>> jingsonglee0@gmail.com
> >>>> > > >> > > >>>>>>>>>>>>>>
> >>>> > > >> > > >>>>>>>>>>>>>>>>> wrote:
> >>>> > > >> > > >>>>>>>>>>>>>>>>>
> >>>> > > >> > > >>>>>>>>>>>>>>>>>> +1 to backport the FLIP-27 adjustments
> to
> >>>> > > >> > > >>> 1.11.x.
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>
> >>>> > > >> > > >>>>>>>>>>>>>>>>>> If possible, that would be great. Many
> >>>> people
> >>>> > > >> > > >>> are
> >>>> > > >> > > >>>>>>> looking
> >>>> > > >> > > >>>>>>>>>>> forward
> >>>> > > >> > > >>>>>>>>>>>>> to
> >>>> > > >> > > >>>>>>>>>>>>>>> the
> >>>> > > >> > > >>>>>>>>>>>>>>>>>> FLIP-27 interface, but they don't want
> to
> >>>> take
> >>>> > > >> > > >>> the
> >>>> > > >> > > >>>>>> risk
> >>>> > > >> > > >>>>>>> to
> >>>> > > >> > > >>>>>>>>>>>> upgrade
> >>>> > > >> > > >>>>>>>>>>>>> to
> >>>> > > >> > > >>>>>>>>>>>>>>>>> 1.12
> >>>> > > >> > > >>>>>>>>>>>>>>>>>> (And wait 1.12). After all, 1.11 is a
> >>>> > > >> > > >> relatively
> >>>> > > >> > > >>>>>> stable
> >>>> > > >> > > >>>>>>>>>>> version.
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>
> >>>> > > >> > > >>>>>>>>>>>>>>>>>> Best,
> >>>> > > >> > > >>>>>>>>>>>>>>>>>> Jingsong
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>
> >>>> > > >> > > >>>>>>>>>>>>>>>>>> On Thu, Oct 29, 2020 at 1:24 AM Stephan
> >>>> Ewen <
> >>>> > > >> > > >>>>>>>>>> sewen@apache.org
> >>>> > > >> > > >>>>>>>>>>>>
> >>>> > > >> > > >>>>>>>>>>>>>>> wrote:
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>> Thanks for starting this.
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>>
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>> +1 form my side to backport the FLIP-27
> >>>> > > >> > > >>>>> adjustments
> >>>> > > >> > > >>>>>> to
> >>>> > > >> > > >>>>>>>>>>> 1.11.x.
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>>
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>> There were quite a few changes, and I
> >>>> think
> >>>> > > >> > > >> we
> >>>> > > >> > > >>>>> need
> >>>> > > >> > > >>>>>> to
> >>>> > > >> > > >>>>>>>>>>>>> cherry-pick
> >>>> > > >> > > >>>>>>>>>>>>>>>>> them
> >>>> > > >> > > >>>>>>>>>>>>>>>>>> all
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>> to not get some inconsistent mix of
> >>>> changes
> >>>> > > >> > > >>> and
> >>>> > > >> > > >>>>> many
> >>>> > > >> > > >>>>>>>>> merge
> >>>> > > >> > > >>>>>>>>>>>>>>> conflicts.
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>> I made a list below of what we need to
> >>>> add
> >>>> > > >> > > >> to
> >>>> > > >> > > >>>>>>>>>> "release-1.11".
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>>
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>> *#### Core Source API Changes to
> backport
> >>>> > > >> > > >> (in
> >>>> > > >> > > >>>>>> REVERSE
> >>>> > > >> > > >>>>>>>>>> order)*
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>>
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>>  (Use: "git log
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>>
> >>>> > > >> > > >>>>>>>>>>>>
> >>>> > > >> > > >>>>>
> >>>> > > flink-core/src/main/java/org/apache/flink/api/connector/source")
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>>
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>> commit
> >>>> > > >> > > >>> 162c072e9265a7b6dd9d6f5459eb7974091c4c4e
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>> [FLINK-19492][core] Consolidate Source
> >>>> > > >> > > >> Events
> >>>> > > >> > > >>>>>> between
> >>>> > > >> > > >>>>>>>>>> Source
> >>>> > > >> > > >>>>>>>>>>>> API
> >>>> > > >> > > >>>>>>>>>>>>>>> and
> >>>> > > >> > > >>>>>>>>>>>>>>>>>> Split
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>> Reader API
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>>
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>> commit
> >>>> > > >> > > >>> 428fbbad85379f71a9fbf1a1ab3e3dcfa9006436
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>> [FLINK-19205][core] Add access to
> >>>> > > >> > > >>> configuration
> >>>> > > >> > > >>>>> and
> >>>> > > >> > > >>>>>>>>>> hostname
> >>>> > > >> > > >>>>>>>>>>> in
> >>>> > > >> > > >>>>>>>>>>>>> the
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>> SourceReaderContext
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>>
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>> commit
> >>>> > > >> > > >>> 723e1790744ddc78d19c7c978442af1383f38d33
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>> [hotfix][core] Add to Source Enumerator
> >>>> > > >> > > >>>>> convenience
> >>>> > > >> > > >>>>>>>>> methods
> >>>> > > >> > > >>>>>>>>>>> to
> >>>> > > >> > > >>>>>>>>>>>>>>> assign
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>> single split
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>>
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>> *#### Connector Base Changes to
> Backport
> >>>> (in
> >>>> > > >> > > >>>>> REVERSE
> >>>> > > >> > > >>>>>>>>>> order)*
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>>
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>>  (Use: "git log
> >>>> > > >> > > >>>>>>> flink-connectors/flink-connector-base")
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>>
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>> commit
> >>>> > > >> > > >>> 401f56fe9d6b0271260edf9787cdcbfe4d03874d
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>> [FLINK-19427][FLINK-19489][tests] Fix
> >>>> test
> >>>> > > >> > > >>>>>> conditions
> >>>> > > >> > > >>>>>>>>> for
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>>
> >>>> > > >> > > >>>>>>>
> >>>> 'SplitFetcherTest.testNotifiesWhenGoingIdleConcurrent()'
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>>
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>> commit
> >>>> > > >> > > >>> 68c5c2ff779d82a1ff81ffaf60d8a1b283797db1
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>> [FLINK-19448][connector base]
> Explicitly
> >>>> > > >> > > >> check
> >>>> > > >> > > >>>> for
> >>>> > > >> > > >>>>>>>>>>> un-expected
> >>>> > > >> > > >>>>>>>>>>>>>>>>> condition
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>> that would leave an inconsistent state
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>>
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>> commit
> >>>> > > >> > > >>> 162c072e9265a7b6dd9d6f5459eb7974091c4c4e
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>> [FLINK-19492][core] Consolidate Source
> >>>> > > >> > > >> Events
> >>>> > > >> > > >>>>>> between
> >>>> > > >> > > >>>>>>>>>> Source
> >>>> > > >> > > >>>>>>>>>>>> API
> >>>> > > >> > > >>>>>>>>>>>>>>> and
> >>>> > > >> > > >>>>>>>>>>>>>>>>>> Split
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>> Reader API
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>>
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>> commit
> >>>> > > >> > > >>> c1ca7a4c7c21ec8868c14cdddd43c559625b794c
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>> [refactor][tests] Move some source test
> >>>> > > >> > > >> utils
> >>>> > > >> > > >>>> from
> >>>> > > >> > > >>>>>>>>>>>>>>>>> flink-connector-base
> >>>> > > >> > > >>>>>>>>>>>>>>>>>> to
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>> flink-core
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>>
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>> commit
> >>>> > > >> > > >>> ee5c4c211c35c70d28252363bbc8400453609977
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>> [FLINK-19251][connectors] Avoid
> confusing
> >>>> > > >> > > >>> queue
> >>>> > > >> > > >>>>>>>>> handling in
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>> "SplitReader.handleSplitsChanges()"
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>>
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>> commit
> >>>> > > >> > > >>> 5abef56b2bf85bcac786f6b16b6899b6cced7176
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>> [FLINK-19250][connectors] Fix error
> >>>> > > >> > > >>> propagation
> >>>> > > >> > > >>>> in
> >>>> > > >> > > >>>>>>>>>> connector
> >>>> > > >> > > >>>>>>>>>>>> base
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>> (SplitFetcherManager).
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>>
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>> commit
> >>>> > > >> > > >>> 8fcca837c55a9216595ee4c03038b52747098dbb
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>> [hotfix][connectors] Improve JavaDocs
> for
> >>>> > > >> > > >>>>>>>>>>>>>>> SingleThreadFetcherManager
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>>
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>> commit
> >>>> > > >> > > >>> 4700bb5dde3303cbe98882f6beb7379425717b01
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>> [FLINK-19225][connectors] Various small
> >>>> > > >> > > >>>>> improvements
> >>>> > > >> > > >>>>>>> to
> >>>> > > >> > > >>>>>>>>>>>>>>>>> SourceReaderBase
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>> (part 2)
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>>
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>> commit
> >>>> > > >> > > >>> 12261c6b7ed6478a9b9f6a69cb58246b83cab9b7
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>> [FLINK-17393][connectors] (follow-up)
> >>>> Wakeup
> >>>> > > >> > > >>> the
> >>>> > > >> > > >>>>>>>>>>> SplitFetchers
> >>>> > > >> > > >>>>>>>>>>>>> more
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>> elegantly.
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>>
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>> commit
> >>>> > > >> > > >>> c60aaff0249bfd6b5871b7f82e03efc487a54d6b
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>> [hotfix][tests] Extend test coverage
> for
> >>>> > > >> > > >>>>>>>>>>>>>>> FutureCompletingBlockingQueue
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>>
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>> commit
> >>>> > > >> > > >>> cef8a587d7fd2fe64cc644da5ed095d82e46f631
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>> [FLINK-19245][connectors] Set default
> >>>> > > >> > > >> capacity
> >>>> > > >> > > >>>> for
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>> FutureCompletingBlockingQueue.
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>>
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>> commit
> >>>> > > >> > > >>> 4ea95782b4c6a2538153d4d16ad3f4839c7de0fb
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>> [FLINK-19223][connectors] Simplify
> >>>> > > >> > > >>> Availability
> >>>> > > >> > > >>>>>> Future
> >>>> > > >> > > >>>>>>>>>> Model
> >>>> > > >> > > >>>>>>>>>>> in
> >>>> > > >> > > >>>>>>>>>>>>>>> Base
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>> Connector
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>>
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>> commit
> >>>> > > >> > > >>> 511857049ba30c8ff0ee56da551fa4a479dc583e
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>> [FLINK-18128][connectors] Ensure idle
> >>>> split
> >>>> > > >> > > >>>>> fetchers
> >>>> > > >> > > >>>>>>>>> lead
> >>>> > > >> > > >>>>>>>>>> to
> >>>> > > >> > > >>>>>>>>>>>>>>>>> availability
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>> notifications.
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>>
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>> commit
> >>>> > > >> > > >>> a8206467af0830dcb89623ea068b5ca3b3450c92
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>> [refactor][core] Eagerly initialize the
> >>>> > > >> > > >>>> FetchTask
> >>>> > > >> > > >>>>> to
> >>>> > > >> > > >>>>>>>>>> support
> >>>> > > >> > > >>>>>>>>>>>>> proper
> >>>> > > >> > > >>>>>>>>>>>>>>>>> unit
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>> testing
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>>
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>> commit
> >>>> > > >> > > >>> 3b2f54bcb437f98e6137c904045cc51072b5c06b
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>> [hotfix][tests] Move constants in
> >>>> > > >> > > >>>> SplitFetcherTest
> >>>> > > >> > > >>>>>>>>> relevant
> >>>> > > >> > > >>>>>>>>>>> to
> >>>> > > >> > > >>>>>>>>>>>>> only
> >>>> > > >> > > >>>>>>>>>>>>>>>>> one
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>> test into test method
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>>
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>> commit
> >>>> > > >> > > >>> d7625760a75a508bf05bcddc380bb4d62ee1743e
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>> [FLINK-19225][connectors] Various small
> >>>> > > >> > > >>>>> improvements
> >>>> > > >> > > >>>>>>> to
> >>>> > > >> > > >>>>>>>>>>>>>>>>> SourceReaderBase
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>>
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>> commit
> >>>> > > >> > > >>> a5b0d3297748c1be47ad579a88f24df2255a8df1
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>> [FLINK-17393][connectors] Wakeup the
> >>>> > > >> > > >>>> SplitFetchers
> >>>> > > >> > > >>>>>>> more
> >>>> > > >> > > >>>>>>>>>>>>> elegantly.
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>>
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>> commit
> >>>> > > >> > > >>> f42a3ebc3e81a034b7221a803c153636fef34903
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>> [FLINK-18680][connectors] Make
> connector
> >>>> > > >> > > >> base
> >>>> > > >> > > >>>>>>>>>>>> RecordsWithSplitIds
> >>>> > > >> > > >>>>>>>>>>>>>>> more
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>> lightweight.
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>>
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>> commit
> >>>> > > >> > > >>> e3d273de822b085183d09b275a445879ff94b350
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>> [FLINK-19162][connectors] Add
> >>>> 'recycle()' to
> >>>> > > >> > > >>> the
> >>>> > > >> > > >>>>>>>>>>>>>>> RecordsWithSplitIds
> >>>> > > >> > > >>>>>>>>>>>>>>>>> to
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>> support reuse of heavy objects.
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>>
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>> commit
> >>>> > > >> > > >>> 8ebc464c2520453a70001cd712abc8dee6ee89e0
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>> [hotfix][testing] Add a set of
> >>>> > > >> > > >> parameterizable
> >>>> > > >> > > >>>>>> testing
> >>>> > > >> > > >>>>>>>>>> mocks
> >>>> > > >> > > >>>>>>>>>>>> for
> >>>> > > >> > > >>>>>>>>>>>>>>> the
> >>>> > > >> > > >>>>>>>>>>>>>>>>>> Split
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>> Reader API
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>>
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>> commit
> >>>> > > >> > > >>> 930a07438be1185388d7150640f294dfe2a5d378
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>> [hotfix][connectors] Add
> >>>> RequestSplitEvent
> >>>> > > >> > > >> to
> >>>> > > >> > > >>>>>>>>>>>>>>> 'flink-connector-base'
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>>
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>> commit
> >>>> > > >> > > >>> e945ce8a933bc378844782f784ca473c767ca159
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>> [refactor][connectors] Backport of the
> >>>> > > >> > > >>>>>> connector-base
> >>>> > > >> > > >>>>>>>>>>> exception
> >>>> > > >> > > >>>>>>>>>>>>>>>>> handling
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>> from the Kafka Connector Pull Request
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>>
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>>
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>> On Wed, Oct 28, 2020 at 5:49 PM Steven
> >>>> Wu <
> >>>> > > >> > > >>>>>>>>>>>> stevenz3wu@gmail.com>
> >>>> > > >> > > >>>>>>>>>>>>>>>>> wrote:
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>>
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>>> I would love to see this FLIP-27
> source
> >>>> > > >> > > >>>>> interface
> >>>> > > >> > > >>>>>>>>>>> improvement
> >>>> > > >> > > >>>>>>>>>>>>> [1]
> >>>> > > >> > > >>>>>>>>>>>>>>>>> made
> >>>> > > >> > > >>>>>>>>>>>>>>>>>> to
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>>> 1.11.3.
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>>>
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>>> [1]
> >>>> > > >> > > >>>>>>> https://issues.apache.org/jira/browse/FLINK-19698
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>>>
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>>> On Wed, Oct 28, 2020 at 12:32 AM
> Tzu-Li
> >>>> > > >> > > >>>> (Gordon)
> >>>> > > >> > > >>>>>>> Tai <
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>> tzulitai@apache.org>
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>>> wrote:
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>>>
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>>>> Thanks for the replies so far!
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>>>> Just to provide a brief update on the
> >>>> > > >> > > >>> status
> >>>> > > >> > > >>>>> of
> >>>> > > >> > > >>>>>>>>>> blockers
> >>>> > > >> > > >>>>>>>>>>>> for
> >>>> > > >> > > >>>>>>>>>>>>>>>>> 1.11.3
> >>>> > > >> > > >>>>>>>>>>>>>>>>>> so
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>>> far:
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>>>> *PR opened, pending reviewer*-
> >>>> > > >> > > >>> [FLINK-19717]
> >>>> > > >> > > >>>>>>>>>>>>>>>>>> SourceReaderBase.pollNext
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>>> may
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>>>> return END_OF_INPUT if
> >>>> SplitReader.fetch
> >>>> > > >> > > >>>>> throws
> >>>> > > >> > > >>>>>> (
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>
> >>>> > > >> > > >>> https://github.com/apache/flink/pull/13776)
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>>>> *PR opened, reviewed + close to being
> >>>> > > >> > > >>>> merged*
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>>>> - [FLINK-19741] Timer service should
> >>>> > > >> > > >> skip
> >>>> > > >> > > >>>>>>> restoring
> >>>> > > >> > > >>>>>>>>>> from
> >>>> > > >> > > >>>>>>>>>>>> raw
> >>>> > > >> > > >>>>>>>>>>>>>>> keyed
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>> stream
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>>>> if it isn't the writer (
> >>>> > > >> > > >>>>>>>>>>>>>>>>>
> https://github.com/apache/flink/pull/13761
> >>>> )
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>>>> - [FLINK-19748] Raw keyed stream key
> >>>> > > >> > > >> group
> >>>> > > >> > > >>>>>>> iterator
> >>>> > > >> > > >>>>>>>>>>> should
> >>>> > > >> > > >>>>>>>>>>>> be
> >>>> > > >> > > >>>>>>>>>>>>>>>>>> skipping
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>>>> unwritten key groups (
> >>>> > > >> > > >>>>>>>>>>>>>>> https://github.com/apache/flink/pull/13772
> )
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>>>> *Merged*
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>>>> - [FLINK-19154] Application mode
> >>>> deletes
> >>>> > > >> > > >>> HA
> >>>> > > >> > > >>>>> data
> >>>> > > >> > > >>>>>>> in
> >>>> > > >> > > >>>>>>>>>> case
> >>>> > > >> > > >>>>>>>>>>> of
> >>>> > > >> > > >>>>>>>>>>>>>>>>> suspended
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>>>> ZooKeeper connection
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>>>> - [FLINK-19569] Upgrade ICU4J to
> 67.1+
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>>>> Right now as it seems, progress is
> >>>> > > >> > > >> mainly
> >>>> > > >> > > >>>>>> blocked
> >>>> > > >> > > >>>>>>>>> on a
> >>>> > > >> > > >>>>>>>>>>>>> reviewer
> >>>> > > >> > > >>>>>>>>>>>>>>>>> for
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>>>> FLINK-19717.
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>>>> Meanwhile, Xintong is keeping an eye
> on
> >>>> > > >> > > >>> test
> >>>> > > >> > > >>>>>>>>>>> instabilities
> >>>> > > >> > > >>>>>>>>>>>>> [1]
> >>>> > > >> > > >>>>>>>>>>>>>>> to
> >>>> > > >> > > >>>>>>>>>>>>>>>>> see
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>> if
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>>>> there are any fixes that should be
> >>>> > > >> > > >> applied
> >>>> > > >> > > >>>> to
> >>>> > > >> > > >>>>>>>>>>>> `release-1.11`.
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>>>> This is also a reminder, that if
> there
> >>>> > > >> > > >> are
> >>>> > > >> > > >>>>> other
> >>>> > > >> > > >>>>>>>>>> blockers
> >>>> > > >> > > >>>>>>>>>>>>> that
> >>>> > > >> > > >>>>>>>>>>>>>>> we
> >>>> > > >> > > >>>>>>>>>>>>>>>>>> need
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>> to
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>>>> be aware of, or a need to
> re-establish
> >>>> > > >> > > >>>>> estimated
> >>>> > > >> > > >>>>>>>>> time
> >>>> > > >> > > >>>>>>>>>> for
> >>>> > > >> > > >>>>>>>>>>>>>>> getting
> >>>> > > >> > > >>>>>>>>>>>>>>>>>> fixes
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>>> in
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>>>> and delay the RC for 1.11.3, please
> do
> >>>> > > >> > > >> let
> >>>> > > >> > > >>>> us
> >>>> > > >> > > >>>>>>> know!
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>>>> Cheers,
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>>>> Gordon
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>>>> [1]
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>>>
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>>
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>
> >>>> > > >> > > >>>>>>>>>>>>>>>>>
> >>>> > > >> > > >>>>>>>>>>>>>>>
> >>>> > > >> > > >>>>>>>>>>>>>
> >>>> > > >> > > >>>>>>>>>>>>
> >>>> > > >> > > >>>>>>>>>>>
> >>>> > > >> > > >>>>>>>>>>
> >>>> > > >> > > >>>>>>>>>
> >>>> > > >> > > >>>>>>>
> >>>> > > >> > > >>>>>>
> >>>> > > >> > > >>>>>
> >>>> > > >> > > >>>>
> >>>> > > >> > > >>>
> >>>> > > >> > > >>
> >>>> > > >> > >
> >>>> > > >> >
> >>>> > > >>
> >>>> > >
> >>>> >
> >>>>
> https://issues.apache.org/jira/issues/?jql=project%20%3D%20FLINK%20AND%20status%20in%20(Open%2C%20%22In%20Progress%22%2C%20Reopened)%20AND%20affectedVersion%20in%20(1.11.0%2C%201.11.1%2C%201.11.2)%20AND%20labels%20%3D%20test-stability
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>>>> On Mon, Oct 26, 2020 at 9:43 PM
> Kostas
> >>>> > > >> > > >>>>> Kloudas <
> >>>> > > >> > > >>>>>>>>>>>>>>>>> kkloudas@apache.org>
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>>>> wrote:
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>> +1 for releasing Flink 1.11.3 as it
> >>>> > > >> > > >>>>> contains a
> >>>> > > >> > > >>>>>>>>> number
> >>>> > > >> > > >>>>>>>>>>> of
> >>>> > > >> > > >>>>>>>>>>>>>>>>> important
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>> fixes and thanks Gordon and Xintong
> >>>> > > >> > > >> for
> >>>> > > >> > > >>>>>>>>> volunteering.
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>> Cheers,
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>> Kostas
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>> On Mon, Oct 26, 2020 at 4:37 AM Yu
> Li
> >>>> > > >> > > >> <
> >>>> > > >> > > >>>>>>>>>>> carp84@gmail.com>
> >>>> > > >> > > >>>>>>>>>>>>>>> wrote:
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>> +1 for releasing Flink 1.11.3, and
> >>>> > > >> > > >>>> thanks
> >>>> > > >> > > >>>>>>> Gordon
> >>>> > > >> > > >>>>>>>>>> and
> >>>> > > >> > > >>>>>>>>>>>>>>> Xintong
> >>>> > > >> > > >>>>>>>>>>>>>>>>> for
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>> volunteering as our release
> >>>> > > >> > > >> managers.
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>> Best Regards,
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>> Yu
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>> On Mon, 26 Oct 2020 at 09:45,
> >>>> > > >> > > >> Xintong
> >>>> > > >> > > >>>>> Song <
> >>>> > > >> > > >>>>>>>>>>>>>>>>>> tonysong820@gmail.com>
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>> wrote:
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>> Thanks Gordan for starting this
> >>>> > > >> > > >>>>>> discussion.
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>> My pleasure to help with the
> >>>> > > >> > > >> release
> >>>> > > >> > > >>>>>>> process.
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>> Thank you~
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>> Xintong Song
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>> On Fri, Oct 23, 2020 at 11:29 PM
> >>>> > > >> > > >>> Till
> >>>> > > >> > > >>>>>>>>> Rohrmann <
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>>> trohrmann@apache.org
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>> wrote:
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>> Thanks for starting this
> >>>> > > >> > > >>> discussion
> >>>> > > >> > > >>>>>>> Gordon.
> >>>> > > >> > > >>>>>>>>>> There
> >>>> > > >> > > >>>>>>>>>>>> are
> >>>> > > >> > > >>>>>>>>>>>>>>> over
> >>>> > > >> > > >>>>>>>>>>>>>>>>>> 100
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>>>> issues
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>> which are fixed for 1.11.3.
> >>>> > > >> > > >> Hence
> >>>> > > >> > > >>> +1
> >>>> > > >> > > >>>>>> for a
> >>>> > > >> > > >>>>>>>>>>> soonish
> >>>> > > >> > > >>>>>>>>>>>>>>> 1.11.3
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>>> release.
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>> Thanks
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>> for volunteering as our release
> >>>> > > >> > > >>>>> managers
> >>>> > > >> > > >>>>>>>>> Gordon
> >>>> > > >> > > >>>>>>>>>>> and
> >>>> > > >> > > >>>>>>>>>>>>>>>>> Xintong!
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>> Cheers,
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>> Till
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>> On Fri, Oct 23, 2020 at 5:02 PM
> >>>> > > >> > > >>>> Tzu-Li
> >>>> > > >> > > >>>>>>>>> (Gordon)
> >>>> > > >> > > >>>>>>>>>>>> Tai <
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>> tzulitai@apache.org
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>> wrote:
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>> Hi,
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>> Xintong and I would like to
> >>>> > > >> > > >>> start a
> >>>> > > >> > > >>>>>>>>> discussion
> >>>> > > >> > > >>>>>>>>>>> for
> >>>> > > >> > > >>>>>>>>>>>>>>>>> releasing
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>>> Flink
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>> 1.11.3
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>> soon.
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>> It seems like we already have a
> >>>> > > >> > > >>> few
> >>>> > > >> > > >>>>>>>>> pressing
> >>>> > > >> > > >>>>>>>>>>>> issues
> >>>> > > >> > > >>>>>>>>>>>>>>> that
> >>>> > > >> > > >>>>>>>>>>>>>>>>>> needs
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>>> to
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>>>> be
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>> included in a new hotfix
> >>>> > > >> > > >> release:
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>>   - Heap-based timers’ restore
> >>>> > > >> > > >>>>>> behaviour
> >>>> > > >> > > >>>>>>>>> is
> >>>> > > >> > > >>>>>>>>>>>>> causing a
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>> critical
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>> recovery
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>>   issue for StateFun [1] [2]
> >>>> > > >> > > >>> [3].
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>>   - There are several
> >>>> > > >> > > >> robustness
> >>>> > > >> > > >>>>>> issues
> >>>> > > >> > > >>>>>>>>> for
> >>>> > > >> > > >>>>>>>>>> the
> >>>> > > >> > > >>>>>>>>>>>>>>> FLIP-27
> >>>> > > >> > > >>>>>>>>>>>>>>>>> new
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>>>> source
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>> API,
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>>   such as [4]. We already have
> >>>> > > >> > > >>>> some
> >>>> > > >> > > >>>>>>> users
> >>>> > > >> > > >>>>>>>>>> using
> >>>> > > >> > > >>>>>>>>>>>> the
> >>>> > > >> > > >>>>>>>>>>>>>>>>> FLIP-27
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>> API
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>> with
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>> 1.11.x,
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>>   so it would be important to
> >>>> > > >> > > >>> get
> >>>> > > >> > > >>>>>> those
> >>>> > > >> > > >>>>>>>>> fixes
> >>>> > > >> > > >>>>>>>>>>> in
> >>>> > > >> > > >>>>>>>>>>>>> for
> >>>> > > >> > > >>>>>>>>>>>>>>>>> 1.11.x
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>> as
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>> well.
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>> Apart from the issues that are
> >>>> > > >> > > >>>>> already
> >>>> > > >> > > >>>>>>>>> marked
> >>>> > > >> > > >>>>>>>>>> as
> >>>> > > >> > > >>>>>>>>>>>>>>> blocker
> >>>> > > >> > > >>>>>>>>>>>>>>>>> for
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>>>> 1.11.3
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>> in
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>> our
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>> JIRA [5], please let us know in
> >>>> > > >> > > >>>> this
> >>>> > > >> > > >>>>>>>>> thread if
> >>>> > > >> > > >>>>>>>>>>>> there
> >>>> > > >> > > >>>>>>>>>>>>>>> is
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>> already
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>> ongoing
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>> work for other important fixes
> >>>> > > >> > > >>> that
> >>>> > > >> > > >>>>> we
> >>>> > > >> > > >>>>>>>>> should
> >>>> > > >> > > >>>>>>>>>>> try
> >>>> > > >> > > >>>>>>>>>>>> to
> >>>> > > >> > > >>>>>>>>>>>>>>>>>> include.
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>> Xintong and I would like to
> >>>> > > >> > > >>>> volunteer
> >>>> > > >> > > >>>>>> for
> >>>> > > >> > > >>>>>>>>>>> managing
> >>>> > > >> > > >>>>>>>>>>>>>>> this
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>> release,
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>>>> and
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>> will
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>> try to communicate the priority
> >>>> > > >> > > >>> of
> >>>> > > >> > > >>>>>>> pending
> >>>> > > >> > > >>>>>>>>>>>> blockers
> >>>> > > >> > > >>>>>>>>>>>>>>> over
> >>>> > > >> > > >>>>>>>>>>>>>>>>> the
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>>> next
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>> few
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>> days.
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>> Since the aforementioned issues
> >>>> > > >> > > >>> are
> >>>> > > >> > > >>>>>> quite
> >>>> > > >> > > >>>>>>>>>>>> critical,
> >>>> > > >> > > >>>>>>>>>>>>>>> we’d
> >>>> > > >> > > >>>>>>>>>>>>>>>>>> like
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>> to
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>>>> aim
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>> for a *feature
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>> freeze by the end of next week
> >>>> > > >> > > >>>> (Oct.
> >>>> > > >> > > >>>>>>> 30th)*
> >>>> > > >> > > >>>>>>>>>> and
> >>>> > > >> > > >>>>>>>>>>>>> start
> >>>> > > >> > > >>>>>>>>>>>>>>> the
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>>> release
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>> voting
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>> process the week after.
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>> If that is too short of a
> >>>> > > >> > > >> notice
> >>>> > > >> > > >>>> and
> >>>> > > >> > > >>>>>> you
> >>>> > > >> > > >>>>>>>>> might
> >>>> > > >> > > >>>>>>>>>>>> need
> >>>> > > >> > > >>>>>>>>>>>>>>> more
> >>>> > > >> > > >>>>>>>>>>>>>>>>>> time,
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>> please
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>> let
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>> us know!
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>> Cheers,
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>> Gordon
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>> [1]
> >>>> > > >> > > >>>>>>>>>>>>>
> >>>> https://issues.apache.org/jira/browse/FLINK-19692
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>> [2]
> >>>> > > >> > > >>>>>>>>>>>>>
> >>>> https://issues.apache.org/jira/browse/FLINK-19741
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>> [3]
> >>>> > > >> > > >>>>>>>>>>>>>
> >>>> https://issues.apache.org/jira/browse/FLINK-19748
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>> [4]
> >>>> > > >> > > >>>>>>>>>>>>>
> >>>> https://issues.apache.org/jira/browse/FLINK-19717
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>> [5]
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>>>
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>>
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>
> >>>> > > >> > > >>>>>>>>>>>>>>>>>
> >>>> > > >> > > >>>>>>>>>>>>>>>
> >>>> > > >> > > >>>>>>>>>>>>>
> >>>> > > >> > > >>>>>>>>>>>>
> >>>> > > >> > > >>>>>>>>>>>
> >>>> > > >> > > >>>>>>>>>>
> >>>> > > >> > > >>>>>>>>>
> >>>> > > >> > > >>>>>>>
> >>>> > > >> > > >>>>>>
> >>>> > > >> > > >>>>>
> >>>> > > >> > > >>>>
> >>>> > > >> > > >>>
> >>>> > > >> > > >>
> >>>> > > >> > >
> >>>> > > >> >
> >>>> > > >>
> >>>> > >
> >>>> >
> >>>>
> https://issues.apache.org/jira/issues/?jql=project%20%3D%20FLINK%20AND%20priority%20%3D%20Blocker%20AND%20fixVersion%20%3D%201.11.3
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>>>
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>>
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>
> >>>> > > >> > > >>>>>>>>>>>>>>>>>> --
> >>>> > > >> > > >>>>>>>>>>>>>>>>>> Best, Jingsong Lee
> >>>> > > >> > > >>>>>>>>>>>>>>>>>>
> >>>> > > >> > > >>>>>>>>>>>>>>>>>
> >>>> > > >> > > >>>>>>>>>>>>>>>>
> >>>> > > >> > > >>>>>>>>>>>>>>>
> >>>> > > >> > > >>>>>>>>>>>>>>
> >>>> > > >> > > >>>>>>>>>>>>>
> >>>> > > >> > > >>>>>>>>>>>>
> >>>> > > >> > > >>>>>>>>>>>
> >>>> > > >> > > >>>>>>>>>>
> >>>> > > >> > > >>>>>>>>>
> >>>> > > >> > > >>>>>>>>
> >>>> > > >> > > >>>>>>>
> >>>> > > >> > > >>>>>>
> >>>> > > >> > > >>>>>
> >>>> > > >> > > >>>>
> >>>> > > >> > > >>>
> >>>> > > >> > > >>
> >>>> > > >> > >
> >>>> > > >> > >
> >>>> > > >> > >
> >>>> > > >> >
> >>>> > > >>
> >>>> > > >
> >>>> > >
> >>>> >
> >>>>
> >>>
>

Re: [DISCUSS] Releasing Apache Flink 1.11.3

Posted by Xintong Song <to...@gmail.com>.
Hi devs,

There has been a bit more delay but we finally get all bockers resolved.

I'm creating the RC right now (commit id:
47c2d97499ed95a33c071e916eb8bde7b179e879).
This could take some time given that I'm doing this for the first time and
accessing to Apache Nexus Repository from my place is slow.

Once the RC is created, I'll start a new voting thread.

Thank you~

Xintong Song



On Fri, Dec 4, 2020 at 11:46 AM Xintong Song <to...@gmail.com> wrote:

> Updates on the release progress.
>
> - FLIP-27 is stabilizing on the master branch. Becket is working on the
> final backportings, which is estimated to be finished this weekend.
> - There's a new blocker FLINK-20442, a license problem discovered during
> the 1.12.0 RC2 testings, which is already fixed for 1.12.
>
> We are hoping to create a RC next Monday, if things go well for the above
> issues.
>
>
> Thank you~
>
> Xintong Song
>
>
>
> On Fri, Nov 27, 2020 at 9:49 AM Xintong Song <to...@gmail.com>
> wrote:
>
>> Hi devs,
>>
>> Updates on the release progress.
>>
>> There are a couple of new blockers emerged since the last update, all of
>> which have been resolved in the past week.
>>
>> The release 1.11.3 is still blocked on FLIP-27.
>> Stephan and Becket are trying to first get the feature stabilized on the
>> master branch for release 1.12. After that, the remaining changes will be
>> backported to 1.11, and the release 1.11.3 will be unblocked.
>>
>> We are hoping to deliver the first RC by the end of next week. During
>> this time, if there's any other things that you believe should be included
>> in this release, please reach out to either me or Gordon.
>>
>> Sorry for the delay.
>>
>> Thank you~
>>
>> Xintong Song
>>
>>
>>
>> On Mon, Nov 23, 2020 at 2:03 PM Xintong Song <to...@gmail.com>
>> wrote:
>>
>>> Thanks for the notice, Zhu.
>>> Let's have the discussion on the jira ticket.
>>>
>>> Thank you~
>>>
>>> Xintong Song
>>>
>>>
>>>
>>> On Mon, Nov 23, 2020 at 12:10 PM Zhu Zhu <re...@gmail.com> wrote:
>>>
>>>> Hi Xintong,
>>>> We just found an issue FLINK-20285 and I think it is a blocker for
>>>> 1.11.3.
>>>>
>>>> Thanks,
>>>> Zhu
>>>>
>>>> Gyula Fóra <gy...@gmail.com> 于2020年11月18日周三 下午9:36写道：
>>>>
>>>> > Hi All!
>>>> >
>>>> > I have found the following issue today which might be considered a
>>>> blocker
>>>> > for this release as well:
>>>> >
>>>> > https://issues.apache.org/jira/browse/FLINK-20221
>>>> >
>>>> > Could someone please quickly provide a second set of eyes and confirm
>>>> that
>>>> > this is indeed a big problem? :)
>>>> >
>>>> > Thank you!
>>>> > Gyula
>>>> >
>>>> > On Wed, Nov 18, 2020 at 5:12 AM Xintong Song <to...@gmail.com>
>>>> > wrote:
>>>> >
>>>> > > Hi devs,
>>>> > >
>>>> > > Updates on the progress of preparing the 1.11.3 release.
>>>> > >
>>>> > > We are approaching the creation of our first release candidate.
>>>> Thanks
>>>> > all
>>>> > > for the efforts so far.
>>>> > >
>>>> > > # Ongoing threads
>>>> > >
>>>> > >    - *FLIP-27 backportings:* 74 out of 102 commits are already
>>>> > backported.
>>>> > >    Stephan and Becket are actively working on this.
>>>> > >    - *Blockers:* There are 2 remaining blockers, both have been
>>>> fixed on
>>>> > >    the master branch and should be backported along with the FLIP-27
>>>> > > changes.
>>>> > >
>>>> > > As soon as the above threads are done, we will create our first
>>>> release
>>>> > > candidate. If there's any other issues that you believe should be a
>>>> > release
>>>> > > blocker, please reach out to either me or Gordon.
>>>> > >
>>>> > > # JIRA version
>>>> > >
>>>> > > Version 1.11.4 has been created on JIRA. I've already tagged some
>>>> of the
>>>> > > tickets with the new fix version. Please feel free to continue with
>>>> the
>>>> > > efforts and merge once they are ready. I'll double check and update
>>>> the
>>>> > fix
>>>> > > versions before the release.
>>>> > >
>>>> > > Thank you~
>>>> > >
>>>> > > Xintong Song
>>>> > >
>>>> > >
>>>> > >
>>>> > > On Thu, Nov 12, 2020 at 1:31 PM Xintong Song <tonysong820@gmail.com
>>>> >
>>>> > > wrote:
>>>> > >
>>>> > > > Thanks for the notice and fix, Roman.
>>>> > > >
>>>> > > > Thank you~
>>>> > > >
>>>> > > > Xintong Song
>>>> > > >
>>>> > > >
>>>> > > >
>>>> > > > On Wed, Nov 11, 2020 at 5:53 PM Khachatryan Roman <
>>>> > > > khachatryan.roman@gmail.com> wrote:
>>>> > > >
>>>> > > >> Hi,
>>>> > > >>
>>>> > > >> I'd like FLINK-20079 [1] to be merged into 1.11 and included in
>>>> > 1.11.3.
>>>> > > >>
>>>> > > >> [1] https://issues.apache.org/jira/browse/FLINK-20079
>>>> > > >>
>>>> > > >> Regards,
>>>> > > >> Roman
>>>> > > >>
>>>> > > >>
>>>> > > >> On Tue, Nov 10, 2020 at 8:21 AM Xintong Song <
>>>> tonysong820@gmail.com>
>>>> > > >> wrote:
>>>> > > >>
>>>> > > >> > Thanks for the notice, Dian.
>>>> > > >> >
>>>> > > >> > Thank you~
>>>> > > >> >
>>>> > > >> > Xintong Song
>>>> > > >> >
>>>> > > >> >
>>>> > > >> >
>>>> > > >> > On Tue, Nov 10, 2020 at 10:19 AM Dian Fu <
>>>> dian0511.fu@gmail.com>
>>>> > > wrote:
>>>> > > >> >
>>>> > > >> > > Hi Xintong,
>>>> > > >> > >
>>>> > > >> > > I want to bring one more issue to your attention [1]. The
>>>> test
>>>> > case
>>>> > > >> > > UnalignedCheckpointCompatibilityITCase.test failed several
>>>> times
>>>> > in
>>>> > > >> the
>>>> > > >> > > last nightly test of release-1.11. We need to figure out if
>>>> it's
>>>> > > just
>>>> > > >> an
>>>> > > >> > > instable test or caused by recent changes.
>>>> > > >> > >
>>>> > > >> > > [1] https://issues.apache.org/jira/browse/FLINK-20065
>>>> > > >> > >
>>>> > > >> > > > 在 2020年11月10日，上午9:24，Xintong Song <to...@gmail.com>
>>>> 写道：
>>>> > > >> > > >
>>>> > > >> > > > Thanks for the replies.
>>>> > > >> > > >
>>>> > > >> > > > Thank you~
>>>> > > >> > > >
>>>> > > >> > > > Xintong Song
>>>> > > >> > > >
>>>> > > >> > > >
>>>> > > >> > > >
>>>> > > >> > > > On Tue, Nov 10, 2020 at 1:09 AM Becket Qin <
>>>> > becket.qin@gmail.com>
>>>> > > >> > wrote:
>>>> > > >> > > >
>>>> > > >> > > >> Hi Xintong,
>>>> > > >> > > >>
>>>> > > >> > > >> Thanks for driving the release. Just want to sync up on
>>>> the
>>>> > > FLIP-27
>>>> > > >> > > >> backporting. Stephan and I are still trying to backport a
>>>> bunch
>>>> > > of
>>>> > > >> > > patches
>>>> > > >> > > >> of Source to 1.11.3. Including:
>>>> > > >> > > >>
>>>> > > >> > > >> [FLINK-19698][connector/common] Add a close() method to
>>>> the
>>>> > > >> > SplitReader.
>>>> > > >> > > >> [FLINK-19717] SourceReaderBase.pollNext may return
>>>> END_OF_INPUT
>>>> > > if
>>>> > > >> > > >> SplitReader.fetch throws
>>>> > > >> > > >> [FLINK-19535] [connector/common] Avoid failing a job
>>>> multiple
>>>> > > >> times in
>>>> > > >> > > >> SourceCoordinator.
>>>> > > >> > > >> [FLINK-19265] [FLINK-20049][core] Source API final
>>>> adjustments.
>>>> > > >> > > >>
>>>> > > >> > > >> and a few more fixes.
>>>> > > >> > > >>
>>>> > > >> > > >> We are currently trying to fix them in 1.12 first so it
>>>> might
>>>> > > take
>>>> > > >> a
>>>> > > >> > > little
>>>> > > >> > > >> longer to backport them to 1.11.3. I think it will
>>>> probably
>>>> > take
>>>> > > >> us a
>>>> > > >> > > few
>>>> > > >> > > >> more days to finish the backport. So that would roughly
>>>> be the
>>>> > > end
>>>> > > >> of
>>>> > > >> > > this
>>>> > > >> > > >> week.
>>>> > > >> > > >>
>>>> > > >> > > >> Thanks,
>>>> > > >> > > >>
>>>> > > >> > > >> Jiangjie (Becket) Qin
>>>> > > >> > > >>
>>>> > > >> > > >>
>>>> > > >> > > >>
>>>> > > >> > > >>
>>>> > > >> > > >> On Mon, Nov 9, 2020 at 9:57 PM Till Rohrmann <
>>>> > > trohrmann@apache.org
>>>> > > >> >
>>>> > > >> > > wrote:
>>>> > > >> > > >>
>>>> > > >> > > >>> Yes, I've downgraded FLINK-19816 to critical.
>>>> > > >> > > >>>
>>>> > > >> > > >>> Cheers,
>>>> > > >> > > >>> Till
>>>> > > >> > > >>>
>>>> > > >> > > >>> On Mon, Nov 9, 2020 at 10:19 AM Xintong Song <
>>>> > > >> tonysong820@gmail.com>
>>>> > > >> > > >>> wrote:
>>>> > > >> > > >>>
>>>> > > >> > > >>>> Thanks for the notice, Till.
>>>> > > >> > > >>>>
>>>> > > >> > > >>>> I just checked and found FLINK-20033 is already fixed.
>>>> Shall
>>>> > we
>>>> > > >> also
>>>> > > >> > > >>>> downgrade FLINK-19816 to `Critical`?
>>>> > > >> > > >>>>
>>>> > > >> > > >>>> Thank you~
>>>> > > >> > > >>>>
>>>> > > >> > > >>>> Xintong Song
>>>> > > >> > > >>>>
>>>> > > >> > > >>>>
>>>> > > >> > > >>>>
>>>> > > >> > > >>>> On Mon, Nov 9, 2020 at 4:42 PM Till Rohrmann <
>>>> > > >> trohrmann@apache.org>
>>>> > > >> > > >>> wrote:
>>>> > > >> > > >>>>
>>>> > > >> > > >>>>> I would like to bring one more critical issue to your
>>>> > > attention
>>>> > > >> > which
>>>> > > >> > > >>> is
>>>> > > >> > > >>>>> FLINK-20033 [1]. I believe that this issue is actually
>>>> > causing
>>>> > > >> what
>>>> > > >> > > >> has
>>>> > > >> > > >>>>> been reported in FLINK-19816 [2]. I hope to have it
>>>> fixed by
>>>> > > the
>>>> > > >> > end
>>>> > > >> > > >> of
>>>> > > >> > > >>>>> today. Once FLINK-20033 is fixed, I think that we
>>>> don't have
>>>> > > to
>>>> > > >> > block
>>>> > > >> > > >>> the
>>>> > > >> > > >>>>> release on FLINK-19816.
>>>> > > >> > > >>>>>
>>>> > > >> > > >>>>> [1] https://issues.apache.org/jira/browse/FLINK-20033
>>>> > > >> > > >>>>> [2] https://issues.apache.org/jira/browse/FLINK-19816
>>>> > > >> > > >>>>>
>>>> > > >> > > >>>>> Cheers,
>>>> > > >> > > >>>>> Till
>>>> > > >> > > >>>>>
>>>> > > >> > > >>>>> On Mon, Nov 9, 2020 at 4:05 AM Xintong Song <
>>>> > > >> tonysong820@gmail.com
>>>> > > >> > >
>>>> > > >> > > >>>> wrote:
>>>> > > >> > > >>>>>
>>>> > > >> > > >>>>>> Hi devs,
>>>> > > >> > > >>>>>>
>>>> > > >> > > >>>>>> I'd like to provide an update on the progress of
>>>> preparing
>>>> > > >> release
>>>> > > >> > > >>>>> 1.11.3.
>>>> > > >> > > >>>>>>
>>>> > > >> > > >>>>>> *Blockers*
>>>> > > >> > > >>>>>> We currently have 3 remaining blockers. (3 resolved
>>>> and 1
>>>> > > >> emerged
>>>> > > >> > > >>>>> compared
>>>> > > >> > > >>>>>> to last week)
>>>> > > >> > > >>>>>>
>>>> > > >> > > >>>>>> - [FLINK-19698] Add close() method and
>>>> > onCheckpointComplete()
>>>> > > >> to
>>>> > > >> > > >> the
>>>> > > >> > > >>>>>> Source.
>>>> > > >> > > >>>>>> The issue has been fixed on the master branch. It's
>>>> > currently
>>>> > > >> > > >> blocked
>>>> > > >> > > >>>> on
>>>> > > >> > > >>>>>> the FLIP-27 backportings to backport it to the 1.11
>>>> branch.
>>>> > > >> > > >>>>>>
>>>> > > >> > > >>>>>> - [FLINK-19717] SourceReaderBase.pollNext may return
>>>> > > >> END_OF_INPUT
>>>> > > >> > > >> if
>>>> > > >> > > >>>>>> SplitReader.fetch throws
>>>> > > >> > > >>>>>> A PR has been opened and reviewed. From the
>>>> discussions on
>>>> > > the
>>>> > > >> PR,
>>>> > > >> > > >> it
>>>> > > >> > > >>>>> looks
>>>> > > >> > > >>>>>> close to mergeable.
>>>> > > >> > > >>>>>>
>>>> > > >> > > >>>>>> - [FLINK-19816] Flink restored from a wrong
>>>> checkpoint (a
>>>> > > very
>>>> > > >> old
>>>> > > >> > > >>> one
>>>> > > >> > > >>>>> and
>>>> > > >> > > >>>>>> not the last completed one)
>>>> > > >> > > >>>>>> This is a newly emerged blocker and Matthias is
>>>> working on
>>>> > > it.
>>>> > > >> > > >>>>>>
>>>> > > >> > > >>>>>> *Test Instabilities*
>>>> > > >> > > >>>>>> We currently have 27 test instabilities[1].
>>>> > > >> > > >>>>>> AFAIK, none of them are as serious as to block the
>>>> 1.11.3
>>>> > > >> release.
>>>> > > >> > > >>>>>>
>>>> > > >> > > >>>>>> *FLIP-27 Backprotings*
>>>> > > >> > > >>>>>>
>>>> > > >> > > >>>>>> I noticed that there's no jira issues opened on the
>>>> FLIP-27
>>>> > > >> > > >>> backporting
>>>> > > >> > > >>>>>> efforts, which is part of the major efforts planned
>>>> for the
>>>> > > >> 1.11.3
>>>> > > >> > > >>>>> release,
>>>> > > >> > > >>>>>> making it hard to track the progress.
>>>> > > >> > > >>>>>>
>>>> > > >> > > >>>>>>
>>>> > > >> > > >>>>>> @Stephan and @Becket, could you please share the
>>>> updates on
>>>> > > the
>>>> > > >> > > >>>>> backporting
>>>> > > >> > > >>>>>> efforts? How is the progress and when are the efforts
>>>> > > expected
>>>> > > >> to
>>>> > > >> > > >> be
>>>> > > >> > > >>>>>> finished? It would be appreciated and helpful if we
>>>> can
>>>> > have
>>>> > > a
>>>> > > >> > jira
>>>> > > >> > > >>>>> ticket
>>>> > > >> > > >>>>>> to track the backporing progress publicly.
>>>> > > >> > > >>>>>>
>>>> > > >> > > >>>>>>
>>>> > > >> > > >>>>>> Thank you~
>>>> > > >> > > >>>>>>
>>>> > > >> > > >>>>>> Xintong Song
>>>> > > >> > > >>>>>>
>>>> > > >> > > >>>>>>
>>>> > > >> > > >>>>>> [1]
>>>> > > >> > > >>>>>>
>>>> > > >> > > >>>>>>
>>>> > > >> > > >>>>>
>>>> > > >> > > >>>>
>>>> > > >> > > >>>
>>>> > > >> > > >>
>>>> > > >> > >
>>>> > > >> >
>>>> > > >>
>>>> > >
>>>> >
>>>> https://issues.apache.org/jira/browse/FLINK-16908?filter=12348580&jql=project%20%3D%20FLINK%20AND%20resolution%20%3D%20Unresolved%20AND%20affectedVersion%20in%20(1.11.0%2C%201.11.1%2C%201.11.2%2C%201.11.3)%20AND%20labels%20%3D%20test-stability%20ORDER%20BY%20updated%20DESC%2C%20created%20DESC
>>>> > > >> > > >>>>>>
>>>> > > >> > > >>>>>> On Wed, Nov 4, 2020 at 3:58 PM Tzu-Li (Gordon) Tai <
>>>> > > >> > > >>>> tzulitai@apache.org>
>>>> > > >> > > >>>>>> wrote:
>>>> > > >> > > >>>>>>
>>>> > > >> > > >>>>>>> Thanks for the efforts so far with pushing for
>>>> 1.11.3.
>>>> > > >> > > >>>>>>>
>>>> > > >> > > >>>>>>> I'd like to provide a mid-week update on how we're
>>>> looking
>>>> > > >> with
>>>> > > >> > > >> all
>>>> > > >> > > >>>> the
>>>> > > >> > > >>>>>>> pending blockers and backports:
>>>> > > >> > > >>>>>>>
>>>> > > >> > > >>>>>>> *Blockers**:*
>>>> > > >> > > >>>>>>>
>>>> > > >> > > >>>>>>> Currently all blockers either have an open PR, or
>>>> have
>>>> > been
>>>> > > >> > > >> merged.
>>>> > > >> > > >>>>> I'll
>>>> > > >> > > >>>>>>> highlight below the blockers *that still require
>>>> reviewing
>>>> > > >> > > >> efforts
>>>> > > >> > > >>>> for
>>>> > > >> > > >>>>>> them
>>>> > > >> > > >>>>>>> to move forward:*
>>>> > > >> > > >>>>>>>
>>>> > > >> > > >>>>>>> - [FLINK-19909] Flink application in attach mode
>>>> could not
>>>> > > >> > > >>> terminate
>>>> > > >> > > >>>>> when
>>>> > > >> > > >>>>>>> the only job is canceled. PR:
>>>> > > >> > > >>>>> https://github.com/apache/flink/pull/13911
>>>> > > >> > > >>>>>>> - [FLINK-19717] SourceReaderBase.pollNext may return
>>>> > > >> END_OF_INPUT
>>>> > > >> > > >>> if
>>>> > > >> > > >>>>>>> SplitReader.fetch throws. PR:
>>>> > > >> > > >>>>> https://github.com/apache/flink/pull/13776
>>>> > > >> > > >>>>>>>
>>>> > > >> > > >>>>>>> The above PRs currently have no reviews at all yet.
>>>> They
>>>> > > seem
>>>> > > >> to
>>>> > > >> > > >>>>> already
>>>> > > >> > > >>>>>>> have designated reviewers.
>>>> > > >> > > >>>>>>>
>>>> > > >> > > >>>>>>>
>>>> > > >> > > >>>>>>> *Backports:*
>>>> > > >> > > >>>>>>> - FLIP-27 robustness improvement backports: Stephan
>>>> is
>>>> > > >> currently
>>>> > > >> > > >>>>> working
>>>> > > >> > > >>>>>> on
>>>> > > >> > > >>>>>>> backporting several FLIP-27 changes. There are no
>>>> PRs yet
>>>> > > for
>>>> > > >> the
>>>> > > >> > > >>>>>>> backported changes.
>>>> > > >> > > >>>>>>> - DataStreamUtils.collect() refactorings backport:
>>>> Steven
>>>> > Wu
>>>> > > >> > > >>>> mentioned
>>>> > > >> > > >>>>> to
>>>> > > >> > > >>>>>>> backport this, but AFAIK this isn't assigned to
>>>> anyone
>>>> > yet.
>>>> > > >> > > >>>>>>>
>>>> > > >> > > >>>>>>> *ETAs*:
>>>> > > >> > > >>>>>>>
>>>> > > >> > > >>>>>>> I'd like to request ETAs for the remaining
>>>> backports, to
>>>> > > >> prevent
>>>> > > >> > > >> a
>>>> > > >> > > >>>>> creep
>>>> > > >> > > >>>>>> in
>>>> > > >> > > >>>>>>> the scope of this bugfix release.
>>>> > > >> > > >>>>>>>
>>>> > > >> > > >>>>>>> We already have Flink users that would benefit from
>>>> fixes
>>>> > > that
>>>> > > >> > > >> have
>>>> > > >> > > >>>>> been
>>>> > > >> > > >>>>>>> merged to release-1.11 already, so technically
>>>> speaking
>>>> > the
>>>> > > >> > > >>> backports
>>>> > > >> > > >>>>>>> should be considered "nice-to-have" (to the best of
>>>> my
>>>> > > >> knowledge
>>>> > > >> > > >> of
>>>> > > >> > > >>>> the
>>>> > > >> > > >>>>>>> changes) and could potentially be moved to a
>>>> follow-up
>>>> > > 1.11.4.
>>>> > > >> > > >>>>>>> Most notably, the Stateful Functions project is
>>>> already
>>>> > > >> waiting
>>>> > > >> > > >> on
>>>> > > >> > > >>>>> Flink
>>>> > > >> > > >>>>>>> 1.11.3 to address critical recovery issues (please
>>>> see the
>>>> > > >> > > >> StateFun
>>>> > > >> > > >>>>> 2.2.1
>>>> > > >> > > >>>>>>> release discussion thread [1]).
>>>> > > >> > > >>>>>>>
>>>> > > >> > > >>>>>>> @Stephan Ewen <se...@apache.org> @Becket Qin <
>>>> > > >> > > >> becket.qin@gmail.com
>>>> > > >> > > >>>>
>>>> > > >> > > >>>>>> could
>>>> > > >> > > >>>>>>> you provide an ETA for the FLIP-27 backports? It
>>>> would
>>>> > help
>>>> > > to
>>>> > > >> > > >> get
>>>> > > >> > > >>> a
>>>> > > >> > > >>>>>> better
>>>> > > >> > > >>>>>>> estimate to decide how we proceed here.
>>>> > > >> > > >>>>>>>
>>>> > > >> > > >>>>>>> Cheers,
>>>> > > >> > > >>>>>>> Gordon
>>>> > > >> > > >>>>>>>
>>>> > > >> > > >>>>>>> [1]
>>>> > > >> > > >>>>>>>
>>>> > > >> > > >>>>>>>
>>>> > > >> > > >>>>>>
>>>> > > >> > > >>>>>
>>>> > > >> > > >>>>
>>>> > > >> > > >>>
>>>> > > >> > > >>
>>>> > > >> > >
>>>> > > >> >
>>>> > > >>
>>>> > >
>>>> >
>>>> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Releasing-StateFun-hotfix-version-2-2-1-td46239.html
>>>> > > >> > > >>>>>>>
>>>> > > >> > > >>>>>>> On Wed, Nov 4, 2020 at 3:16 PM Tzu-Li (Gordon) Tai <
>>>> > > >> > > >>>>> tzulitai@apache.org>
>>>> > > >> > > >>>>>>> wrote:
>>>> > > >> > > >>>>>>>
>>>> > > >> > > >>>>>>>>> The collect() utils can be picked back, I see no
>>>> issue
>>>> > > with
>>>> > > >> > > >>> that
>>>> > > >> > > >>>>> (it
>>>> > > >> > > >>>>>> is
>>>> > > >> > > >>>>>>>> isolated utilities).
>>>> > > >> > > >>>>>>>>
>>>> > > >> > > >>>>>>>> Just checking on all the requested backports
>>>> mentioned in
>>>> > > >> this
>>>> > > >> > > >>>>> thread,
>>>> > > >> > > >>>>>>> and
>>>> > > >> > > >>>>>>>> figuring out which ones seem to still be unassigned
>>>> /
>>>> > open.
>>>> > > >> > > >>>>>>>>
>>>> > > >> > > >>>>>>>> Is someone working on backporting
>>>> > > >> > > >>>>>>>>
>>>> > > >> > > >>>>>>>
>>>> > > >> > > >>>>>>
>>>> > > >> > > >>>>>
>>>> > > >> > > >>>>
>>>> > > >> > > >>>
>>>> > > >> > > >>
>>>> > > >> > >
>>>> > > >> >
>>>> > > >>
>>>> > >
>>>> >
>>>> https://github.com/apache/flink/commit/09a7a66b7313fea64817fe960a8da1265b428efc
>>>> > > >> > > >>>>>>>> to release-1.11 at the moment?
>>>> > > >> > > >>>>>>>>
>>>> > > >> > > >>>>>>>> On Wed, Nov 4, 2020 at 1:39 AM Steven Wu <
>>>> > > >> stevenz3wu@gmail.com
>>>> > > >> > > >>>
>>>> > > >> > > >>>>> wrote:
>>>> > > >> > > >>>>>>>>
>>>> > > >> > > >>>>>>>>> @Stephan Ewen <se...@apache.org> yeah, we can do
>>>> that.
>>>> > > >> don't
>>>> > > >> > > >>>> worry
>>>> > > >> > > >>>>>>> about
>>>> > > >> > > >>>>>>>>> it. your earlier email had the perfect explanation
>>>> on
>>>> > why
>>>> > > >> file
>>>> > > >> > > >>>>> source
>>>> > > >> > > >>>>>>>>> shouldn't be backported.
>>>> > > >> > > >>>>>>>>>
>>>> > > >> > > >>>>>>>>> On Tue, Nov 3, 2020 at 3:37 AM Stephan Ewen <
>>>> > > >> sewen@apache.org
>>>> > > >> > > >>>
>>>> > > >> > > >>>>> wrote:
>>>> > > >> > > >>>>>>>>>
>>>> > > >> > > >>>>>>>>>> @Steven would it be possible to initially copy
>>>> some of
>>>> > > the
>>>> > > >> > > >>> code
>>>> > > >> > > >>>>> into
>>>> > > >> > > >>>>>>> the
>>>> > > >> > > >>>>>>>>>> iceberg source and later replace it by a
>>>> dependency on
>>>> > > the
>>>> > > >> > > >>> Flink
>>>> > > >> > > >>>>>> file
>>>> > > >> > > >>>>>>>>>> source?
>>>> > > >> > > >>>>>>>>>>
>>>> > > >> > > >>>>>>>>>> On Mon, Nov 2, 2020 at 8:33 PM Steven Wu <
>>>> > > >> > > >>> stevenz3wu@gmail.com>
>>>> > > >> > > >>>>>>> wrote:
>>>> > > >> > > >>>>>>>>>>
>>>> > > >> > > >>>>>>>>>>> Stephan, thanks a lot for explaining the file
>>>> > connector.
>>>> > > >> > > >>> that
>>>> > > >> > > >>>>>> makes
>>>> > > >> > > >>>>>>>>>> sense.
>>>> > > >> > > >>>>>>>>>>>
>>>> > > >> > > >>>>>>>>>>> I was asking because we were trying to reuse
>>>> some of
>>>> > the
>>>> > > >> > > >>>>>>>>> implementations
>>>> > > >> > > >>>>>>>>>> in
>>>> > > >> > > >>>>>>>>>>> the file source for Iceberg source. Flink Iceberg
>>>> > source
>>>> > > >> > > >>> lives
>>>> > > >> > > >>>>> in
>>>> > > >> > > >>>>>>> the
>>>> > > >> > > >>>>>>>>>>> Iceberg repo, which is not possible to code
>>>> against
>>>> > the
>>>> > > >> > > >>> master
>>>> > > >> > > >>>>>>> branch
>>>> > > >> > > >>>>>>>>> of
>>>> > > >> > > >>>>>>>>>>> the Flink code.
>>>> > > >> > > >>>>>>>>>>>
>>>> > > >> > > >>>>>>>>>>> On Mon, Nov 2, 2020 at 3:31 AM Stephan Ewen <
>>>> > > >> > > >>> sewen@apache.org
>>>> > > >> > > >>>>>
>>>> > > >> > > >>>>>>> wrote:
>>>> > > >> > > >>>>>>>>>>>
>>>> > > >> > > >>>>>>>>>>>> Hi Steven!
>>>> > > >> > > >>>>>>>>>>>>
>>>> > > >> > > >>>>>>>>>>>> So far there are no plans to pick back the file
>>>> > system
>>>> > > >> > > >>>>> connector
>>>> > > >> > > >>>>>>>>> code.
>>>> > > >> > > >>>>>>>>>>> This
>>>> > > >> > > >>>>>>>>>>>> is still evolving and not finalized for 1.12,
>>>> so I
>>>> > > don't
>>>> > > >> > > >>>> feel
>>>> > > >> > > >>>>> it
>>>> > > >> > > >>>>>>> is
>>>> > > >> > > >>>>>>>>> a
>>>> > > >> > > >>>>>>>>>>> good
>>>> > > >> > > >>>>>>>>>>>> candidate to be backported.
>>>> > > >> > > >>>>>>>>>>>> However, with the base connector changes
>>>> backported,
>>>> > > you
>>>> > > >> > > >>>>> should
>>>> > > >> > > >>>>>> be
>>>> > > >> > > >>>>>>>>> able
>>>> > > >> > > >>>>>>>>>>> to
>>>> > > >> > > >>>>>>>>>>>> run the file connector code from master against
>>>> > 1.11.3.
>>>> > > >> > > >>>>>>>>>>>>
>>>> > > >> > > >>>>>>>>>>>> The collect() utils can be picked back, I see no
>>>> > issue
>>>> > > >> > > >>> with
>>>> > > >> > > >>>>> that
>>>> > > >> > > >>>>>>>>> (it is
>>>> > > >> > > >>>>>>>>>>>> isolated utilities).
>>>> > > >> > > >>>>>>>>>>>>
>>>> > > >> > > >>>>>>>>>>>> Best,
>>>> > > >> > > >>>>>>>>>>>> Stephan
>>>> > > >> > > >>>>>>>>>>>>
>>>> > > >> > > >>>>>>>>>>>>
>>>> > > >> > > >>>>>>>>>>>> On Mon, Nov 2, 2020 at 3:02 AM Steven Wu <
>>>> > > >> > > >>>>> stevenz3wu@gmail.com>
>>>> > > >> > > >>>>>>>>> wrote:
>>>> > > >> > > >>>>>>>>>>>>
>>>> > > >> > > >>>>>>>>>>>>> Basically, it would be great to get the latest
>>>> code
>>>> > in
>>>> > > >> > > >>> the
>>>> > > >> > > >>>>>>>>>>>>> flink-connector-files (FLIP-27).
>>>> > > >> > > >>>>>>>>>>>>>
>>>> > > >> > > >>>>>>>>>>>>> On Sat, Oct 31, 2020 at 9:57 AM Steven Wu <
>>>> > > >> > > >>>>>> stevenz3wu@gmail.com
>>>> > > >> > > >>>>>>>>
>>>> > > >> > > >>>>>>>>>>> wrote:
>>>> > > >> > > >>>>>>>>>>>>>
>>>> > > >> > > >>>>>>>>>>>>>> Stephan, it will be great if we can also
>>>> backport
>>>> > > >> > > >> the
>>>> > > >> > > >>>>>>>>>> DataStreamUtils
>>>> > > >> > > >>>>>>>>>>>>>> related commits that help with collecting
>>>> output
>>>> > > >> > > >> from
>>>> > > >> > > >>>>>>> unbounded
>>>> > > >> > > >>>>>>>>>>>> streams.
>>>> > > >> > > >>>>>>>>>>>>>> e.g.
>>>> > > >> > > >>>>>>>>>>>>>>
>>>> > > >> > > >>>>>>>>>>>>>>
>>>> > > >> > > >>>>>>>>>>>>>>
>>>> > > >> > > >>>>>>>>>>>>>
>>>> > > >> > > >>>>>>>>>>>>
>>>> > > >> > > >>>>>>>>>>>
>>>> > > >> > > >>>>>>>>>>
>>>> > > >> > > >>>>>>>>>
>>>> > > >> > > >>>>>>>
>>>> > > >> > > >>>>>>
>>>> > > >> > > >>>>>
>>>> > > >> > > >>>>
>>>> > > >> > > >>>
>>>> > > >> > > >>
>>>> > > >> > >
>>>> > > >> >
>>>> > > >>
>>>> > >
>>>> >
>>>> https://github.com/apache/flink/commit/09a7a66b7313fea64817fe960a8da1265b428efc
>>>> > > >> > > >>>>>>>>>>>>>>
>>>> > > >> > > >>>>>>>>>>>>>> I tried to copy and paste the code to unblock
>>>> > > >> > > >> myself.
>>>> > > >> > > >>>> but
>>>> > > >> > > >>>>> it
>>>> > > >> > > >>>>>>>>>> quickly
>>>> > > >> > > >>>>>>>>>>>> got
>>>> > > >> > > >>>>>>>>>>>>>> into the rabbit hole of more and more code.
>>>> > > >> > > >>>>>>>>>>>>>>
>>>> > > >> > > >>>>>>>>>>>>>> On Fri, Oct 30, 2020 at 11:02 AM Stephan Ewen
>>>> <
>>>> > > >> > > >>>>>>> sewen@apache.org
>>>> > > >> > > >>>>>>>>>>
>>>> > > >> > > >>>>>>>>>>>> wrote:
>>>> > > >> > > >>>>>>>>>>>>>>
>>>> > > >> > > >>>>>>>>>>>>>>> I have started with backporting the source
>>>> API
>>>> > > >> > > >>> changes.
>>>> > > >> > > >>>>>> Some
>>>> > > >> > > >>>>>>>>> minor
>>>> > > >> > > >>>>>>>>>>>>>>> conflicts to solve, will need a bit more to
>>>> finish
>>>> > > >> > > >>>> this.
>>>> > > >> > > >>>>>>>>>>>>>>>
>>>> > > >> > > >>>>>>>>>>>>>>> On Fri, Oct 30, 2020 at 7:25 AM Tzu-Li
>>>> (Gordon)
>>>> > > >> > > >> Tai <
>>>> > > >> > > >>>>>>>>>>>>> tzulitai@apache.org>
>>>> > > >> > > >>>>>>>>>>>>>>> wrote:
>>>> > > >> > > >>>>>>>>>>>>>>>
>>>> > > >> > > >>>>>>>>>>>>>>>> @Stephan Ewen <se...@apache.org>
>>>> > > >> > > >>>>>>>>>>>>>>>> Are there already plans or ongoing efforts
>>>> for
>>>> > > >> > > >>>>>> backporting
>>>> > > >> > > >>>>>>>>> the
>>>> > > >> > > >>>>>>>>>>> list
>>>> > > >> > > >>>>>>>>>>>> of
>>>> > > >> > > >>>>>>>>>>>>>>>> FLIP-27 changes that you posted?
>>>> > > >> > > >>>>>>>>>>>>>>>>
>>>> > > >> > > >>>>>>>>>>>>>>>> On Thu, Oct 29, 2020 at 7:08 PM Xintong
>>>> Song <
>>>> > > >> > > >>>>>>>>>>> tonysong820@gmail.com
>>>> > > >> > > >>>>>>>>>>>>>
>>>> > > >> > > >>>>>>>>>>>>>>>> wrote:
>>>> > > >> > > >>>>>>>>>>>>>>>>
>>>> > > >> > > >>>>>>>>>>>>>>>>> Hi folks,
>>>> > > >> > > >>>>>>>>>>>>>>>>>
>>>> > > >> > > >>>>>>>>>>>>>>>>> Just to provide some updates concerning the
>>>> > > >> > > >> status
>>>> > > >> > > >>>> on
>>>> > > >> > > >>>>>> the
>>>> > > >> > > >>>>>>>>>>>>>>>>> test instabilities.
>>>> > > >> > > >>>>>>>>>>>>>>>>>
>>>> > > >> > > >>>>>>>>>>>>>>>>> Currently, we have 30 unresolved tickets
>>>> labeled
>>>> > > >> > > >>>> with
>>>> > > >> > > >>>>>>>>> `Affects
>>>> > > >> > > >>>>>>>>>>>>> Version`
>>>> > > >> > > >>>>>>>>>>>>>>>>> 1.11.x.
>>>> > > >> > > >>>>>>>>>>>>>>>>>
>>>> > > >> > > >>>>>>>>>>>>>>>>>
>>>> > > >> > > >>>>>>>>>>>>>>>
>>>> > > >> > > >>>>>>>>>>>>>
>>>> > > >> > > >>>>>>>>>>>>
>>>> > > >> > > >>>>>>>>>>>
>>>> > > >> > > >>>>>>>>>>
>>>> > > >> > > >>>>>>>>>
>>>> > > >> > > >>>>>>>
>>>> > > >> > > >>>>>>
>>>> > > >> > > >>>>>
>>>> > > >> > > >>>>
>>>> > > >> > > >>>
>>>> > > >> > > >>
>>>> > > >> > >
>>>> > > >> >
>>>> > > >>
>>>> > >
>>>> >
>>>> https://issues.apache.org/jira/browse/FLINK-19775?filter=12348580&jql=project%20%3D%20FLINK%20AND%20resolution%20%3D%20Unresolved%20AND%20affectedVersion%20in%20(1.11.0%2C%201.11.1%2C%201.11.2%2C%201.11.3)%20AND%20labels%20%3D%20test-stability%20ORDER%20BY%20created%20DESC
>>>> > > >> > > >>>>>>>>>>>>>>>>>
>>>> > > >> > > >>>>>>>>>>>>>>>>> Among the 30 tickets, 11 of them are:
>>>> > > >> > > >>>>>>>>>>>>>>>>> - Have occured in the recent 3 months
>>>> > > >> > > >>>>>>>>>>>>>>>>> - Not confirmed to be pure testability
>>>> issues
>>>> > > >> > > >>>>>>>>>>>>>>>>> - Not confirmed to be rare condition cases
>>>> > > >> > > >>>>>>>>>>>>>>>>>
>>>> > > >> > > >>>>>>>>>>>>>>>>> It would be nice if someone familiar with
>>>> these
>>>> > > >> > > >>>>>> components
>>>> > > >> > > >>>>>>>>> can
>>>> > > >> > > >>>>>>>>>>>> take a
>>>> > > >> > > >>>>>>>>>>>>>>> look
>>>> > > >> > > >>>>>>>>>>>>>>>>> into these issues.
>>>> > > >> > > >>>>>>>>>>>>>>>>>
>>>> > > >> > > >>>>>>>>>>>>>>>>> -
>>>> > > >> > > >>> https://issues.apache.org/jira/browse/FLINK-17159
>>>> > > >> > > >>>>>> (ES6)
>>>> > > >> > > >>>>>>>>>>>>>>>>> -
>>>> > > >> > > >>> https://issues.apache.org/jira/browse/FLINK-17912
>>>> > > >> > > >>>>>>> (Kafka)
>>>> > > >> > > >>>>>>>>>>>>>>>>> -
>>>> > > >> > > >>> https://issues.apache.org/jira/browse/FLINK-17949
>>>> > > >> > > >>>>>>> (Kafka)
>>>> > > >> > > >>>>>>>>>>>>>>>>> ⁃
>>>> > > >> > > >>> https://issues.apache.org/jira/browse/FLINK-18444
>>>> > > >> > > >>>>>>> (Kafka)
>>>> > > >> > > >>>>>>>>>>>>>>>>> -
>>>> > > >> > > >>> https://issues.apache.org/jira/browse/FLINK-18634
>>>> > > >> > > >>>>>>> (Kafka)
>>>> > > >> > > >>>>>>>>>>>>>>>>> -
>>>> > > >> > > >>> https://issues.apache.org/jira/browse/FLINK-18648
>>>> > > >> > > >>>>>>> (Kafka)
>>>> > > >> > > >>>>>>>>>>>>>>>>> -
>>>> > > >> > > >>> https://issues.apache.org/jira/browse/FLINK-18807
>>>> > > >> > > >>>>>>> (Kafka)
>>>> > > >> > > >>>>>>>>>>>>>>>>> -
>>>> > > >> > > >>> https://issues.apache.org/jira/browse/FLINK-19369
>>>> > > >> > > >>>>>>>>>>>> (BlobClientTest)
>>>> > > >> > > >>>>>>>>>>>>>>>>> -
>>>> > > >> > > >>> https://issues.apache.org/jira/browse/FLINK-19436
>>>> > > >> > > >>>>>>> (TPCDS)
>>>> > > >> > > >>>>>>>>>>>>>>>>> -
>>>> > > >> > > >>> https://issues.apache.org/jira/browse/FLINK-19690
>>>> > > >> > > >>>>>>>>>>>> (Format/Parquet)
>>>> > > >> > > >>>>>>>>>>>>>>>>> -
>>>> > > >> > > >>> https://issues.apache.org/jira/browse/FLINK-19775
>>>> > > >> > > >>>>>>>>>>>>>>>>> (SystemProcessingTimeServiceTest)
>>>> > > >> > > >>>>>>>>>>>>>>>>>
>>>> > > >> > > >>>>>>>>>>>>>>>>> Thank you~
>>>> > > >> > > >>>>>>>>>>>>>>>>>
>>>> > > >> > > >>>>>>>>>>>>>>>>> Xintong Song
>>>> > > >> > > >>>>>>>>>>>>>>>>>
>>>> > > >> > > >>>>>>>>>>>>>>>>>
>>>> > > >> > > >>>>>>>>>>>>>>>>>
>>>> > > >> > > >>>>>>>>>>>>>>>>> On Thu, Oct 29, 2020 at 10:21 AM Jingsong
>>>> Li <
>>>> > > >> > > >>>>>>>>>>>> jingsonglee0@gmail.com
>>>> > > >> > > >>>>>>>>>>>>>>
>>>> > > >> > > >>>>>>>>>>>>>>>>> wrote:
>>>> > > >> > > >>>>>>>>>>>>>>>>>
>>>> > > >> > > >>>>>>>>>>>>>>>>>> +1 to backport the FLIP-27 adjustments to
>>>> > > >> > > >>> 1.11.x.
>>>> > > >> > > >>>>>>>>>>>>>>>>>>
>>>> > > >> > > >>>>>>>>>>>>>>>>>> If possible, that would be great. Many
>>>> people
>>>> > > >> > > >>> are
>>>> > > >> > > >>>>>>> looking
>>>> > > >> > > >>>>>>>>>>> forward
>>>> > > >> > > >>>>>>>>>>>>> to
>>>> > > >> > > >>>>>>>>>>>>>>> the
>>>> > > >> > > >>>>>>>>>>>>>>>>>> FLIP-27 interface, but they don't want to
>>>> take
>>>> > > >> > > >>> the
>>>> > > >> > > >>>>>> risk
>>>> > > >> > > >>>>>>> to
>>>> > > >> > > >>>>>>>>>>>> upgrade
>>>> > > >> > > >>>>>>>>>>>>> to
>>>> > > >> > > >>>>>>>>>>>>>>>>> 1.12
>>>> > > >> > > >>>>>>>>>>>>>>>>>> (And wait 1.12). After all, 1.11 is a
>>>> > > >> > > >> relatively
>>>> > > >> > > >>>>>> stable
>>>> > > >> > > >>>>>>>>>>> version.
>>>> > > >> > > >>>>>>>>>>>>>>>>>>
>>>> > > >> > > >>>>>>>>>>>>>>>>>> Best,
>>>> > > >> > > >>>>>>>>>>>>>>>>>> Jingsong
>>>> > > >> > > >>>>>>>>>>>>>>>>>>
>>>> > > >> > > >>>>>>>>>>>>>>>>>> On Thu, Oct 29, 2020 at 1:24 AM Stephan
>>>> Ewen <
>>>> > > >> > > >>>>>>>>>> sewen@apache.org
>>>> > > >> > > >>>>>>>>>>>>
>>>> > > >> > > >>>>>>>>>>>>>>> wrote:
>>>> > > >> > > >>>>>>>>>>>>>>>>>>
>>>> > > >> > > >>>>>>>>>>>>>>>>>>> Thanks for starting this.
>>>> > > >> > > >>>>>>>>>>>>>>>>>>>
>>>> > > >> > > >>>>>>>>>>>>>>>>>>> +1 form my side to backport the FLIP-27
>>>> > > >> > > >>>>> adjustments
>>>> > > >> > > >>>>>> to
>>>> > > >> > > >>>>>>>>>>> 1.11.x.
>>>> > > >> > > >>>>>>>>>>>>>>>>>>>
>>>> > > >> > > >>>>>>>>>>>>>>>>>>> There were quite a few changes, and I
>>>> think
>>>> > > >> > > >> we
>>>> > > >> > > >>>>> need
>>>> > > >> > > >>>>>> to
>>>> > > >> > > >>>>>>>>>>>>> cherry-pick
>>>> > > >> > > >>>>>>>>>>>>>>>>> them
>>>> > > >> > > >>>>>>>>>>>>>>>>>> all
>>>> > > >> > > >>>>>>>>>>>>>>>>>>> to not get some inconsistent mix of
>>>> changes
>>>> > > >> > > >>> and
>>>> > > >> > > >>>>> many
>>>> > > >> > > >>>>>>>>> merge
>>>> > > >> > > >>>>>>>>>>>>>>> conflicts.
>>>> > > >> > > >>>>>>>>>>>>>>>>>>> I made a list below of what we need to
>>>> add
>>>> > > >> > > >> to
>>>> > > >> > > >>>>>>>>>> "release-1.11".
>>>> > > >> > > >>>>>>>>>>>>>>>>>>>
>>>> > > >> > > >>>>>>>>>>>>>>>>>>> *#### Core Source API Changes to backport
>>>> > > >> > > >> (in
>>>> > > >> > > >>>>>> REVERSE
>>>> > > >> > > >>>>>>>>>> order)*
>>>> > > >> > > >>>>>>>>>>>>>>>>>>>
>>>> > > >> > > >>>>>>>>>>>>>>>>>>>  (Use: "git log
>>>> > > >> > > >>>>>>>>>>>>>>>>>>>
>>>> > > >> > > >>>>>>>>>>>>
>>>> > > >> > > >>>>>
>>>> > > flink-core/src/main/java/org/apache/flink/api/connector/source")
>>>> > > >> > > >>>>>>>>>>>>>>>>>>>
>>>> > > >> > > >>>>>>>>>>>>>>>>>>> commit
>>>> > > >> > > >>> 162c072e9265a7b6dd9d6f5459eb7974091c4c4e
>>>> > > >> > > >>>>>>>>>>>>>>>>>>> [FLINK-19492][core] Consolidate Source
>>>> > > >> > > >> Events
>>>> > > >> > > >>>>>> between
>>>> > > >> > > >>>>>>>>>> Source
>>>> > > >> > > >>>>>>>>>>>> API
>>>> > > >> > > >>>>>>>>>>>>>>> and
>>>> > > >> > > >>>>>>>>>>>>>>>>>> Split
>>>> > > >> > > >>>>>>>>>>>>>>>>>>> Reader API
>>>> > > >> > > >>>>>>>>>>>>>>>>>>>
>>>> > > >> > > >>>>>>>>>>>>>>>>>>> commit
>>>> > > >> > > >>> 428fbbad85379f71a9fbf1a1ab3e3dcfa9006436
>>>> > > >> > > >>>>>>>>>>>>>>>>>>> [FLINK-19205][core] Add access to
>>>> > > >> > > >>> configuration
>>>> > > >> > > >>>>> and
>>>> > > >> > > >>>>>>>>>> hostname
>>>> > > >> > > >>>>>>>>>>> in
>>>> > > >> > > >>>>>>>>>>>>> the
>>>> > > >> > > >>>>>>>>>>>>>>>>>>> SourceReaderContext
>>>> > > >> > > >>>>>>>>>>>>>>>>>>>
>>>> > > >> > > >>>>>>>>>>>>>>>>>>> commit
>>>> > > >> > > >>> 723e1790744ddc78d19c7c978442af1383f38d33
>>>> > > >> > > >>>>>>>>>>>>>>>>>>> [hotfix][core] Add to Source Enumerator
>>>> > > >> > > >>>>> convenience
>>>> > > >> > > >>>>>>>>> methods
>>>> > > >> > > >>>>>>>>>>> to
>>>> > > >> > > >>>>>>>>>>>>>>> assign
>>>> > > >> > > >>>>>>>>>>>>>>>>>>> single split
>>>> > > >> > > >>>>>>>>>>>>>>>>>>>
>>>> > > >> > > >>>>>>>>>>>>>>>>>>> *#### Connector Base Changes to Backport
>>>> (in
>>>> > > >> > > >>>>> REVERSE
>>>> > > >> > > >>>>>>>>>> order)*
>>>> > > >> > > >>>>>>>>>>>>>>>>>>>
>>>> > > >> > > >>>>>>>>>>>>>>>>>>>  (Use: "git log
>>>> > > >> > > >>>>>>> flink-connectors/flink-connector-base")
>>>> > > >> > > >>>>>>>>>>>>>>>>>>>
>>>> > > >> > > >>>>>>>>>>>>>>>>>>> commit
>>>> > > >> > > >>> 401f56fe9d6b0271260edf9787cdcbfe4d03874d
>>>> > > >> > > >>>>>>>>>>>>>>>>>>> [FLINK-19427][FLINK-19489][tests] Fix
>>>> test
>>>> > > >> > > >>>>>> conditions
>>>> > > >> > > >>>>>>>>> for
>>>> > > >> > > >>>>>>>>>>>>>>>>>>>
>>>> > > >> > > >>>>>>>
>>>> 'SplitFetcherTest.testNotifiesWhenGoingIdleConcurrent()'
>>>> > > >> > > >>>>>>>>>>>>>>>>>>>
>>>> > > >> > > >>>>>>>>>>>>>>>>>>> commit
>>>> > > >> > > >>> 68c5c2ff779d82a1ff81ffaf60d8a1b283797db1
>>>> > > >> > > >>>>>>>>>>>>>>>>>>> [FLINK-19448][connector base] Explicitly
>>>> > > >> > > >> check
>>>> > > >> > > >>>> for
>>>> > > >> > > >>>>>>>>>>> un-expected
>>>> > > >> > > >>>>>>>>>>>>>>>>> condition
>>>> > > >> > > >>>>>>>>>>>>>>>>>>> that would leave an inconsistent state
>>>> > > >> > > >>>>>>>>>>>>>>>>>>>
>>>> > > >> > > >>>>>>>>>>>>>>>>>>> commit
>>>> > > >> > > >>> 162c072e9265a7b6dd9d6f5459eb7974091c4c4e
>>>> > > >> > > >>>>>>>>>>>>>>>>>>> [FLINK-19492][core] Consolidate Source
>>>> > > >> > > >> Events
>>>> > > >> > > >>>>>> between
>>>> > > >> > > >>>>>>>>>> Source
>>>> > > >> > > >>>>>>>>>>>> API
>>>> > > >> > > >>>>>>>>>>>>>>> and
>>>> > > >> > > >>>>>>>>>>>>>>>>>> Split
>>>> > > >> > > >>>>>>>>>>>>>>>>>>> Reader API
>>>> > > >> > > >>>>>>>>>>>>>>>>>>>
>>>> > > >> > > >>>>>>>>>>>>>>>>>>> commit
>>>> > > >> > > >>> c1ca7a4c7c21ec8868c14cdddd43c559625b794c
>>>> > > >> > > >>>>>>>>>>>>>>>>>>> [refactor][tests] Move some source test
>>>> > > >> > > >> utils
>>>> > > >> > > >>>> from
>>>> > > >> > > >>>>>>>>>>>>>>>>> flink-connector-base
>>>> > > >> > > >>>>>>>>>>>>>>>>>> to
>>>> > > >> > > >>>>>>>>>>>>>>>>>>> flink-core
>>>> > > >> > > >>>>>>>>>>>>>>>>>>>
>>>> > > >> > > >>>>>>>>>>>>>>>>>>> commit
>>>> > > >> > > >>> ee5c4c211c35c70d28252363bbc8400453609977
>>>> > > >> > > >>>>>>>>>>>>>>>>>>> [FLINK-19251][connectors] Avoid confusing
>>>> > > >> > > >>> queue
>>>> > > >> > > >>>>>>>>> handling in
>>>> > > >> > > >>>>>>>>>>>>>>>>>>> "SplitReader.handleSplitsChanges()"
>>>> > > >> > > >>>>>>>>>>>>>>>>>>>
>>>> > > >> > > >>>>>>>>>>>>>>>>>>> commit
>>>> > > >> > > >>> 5abef56b2bf85bcac786f6b16b6899b6cced7176
>>>> > > >> > > >>>>>>>>>>>>>>>>>>> [FLINK-19250][connectors] Fix error
>>>> > > >> > > >>> propagation
>>>> > > >> > > >>>> in
>>>> > > >> > > >>>>>>>>>> connector
>>>> > > >> > > >>>>>>>>>>>> base
>>>> > > >> > > >>>>>>>>>>>>>>>>>>> (SplitFetcherManager).
>>>> > > >> > > >>>>>>>>>>>>>>>>>>>
>>>> > > >> > > >>>>>>>>>>>>>>>>>>> commit
>>>> > > >> > > >>> 8fcca837c55a9216595ee4c03038b52747098dbb
>>>> > > >> > > >>>>>>>>>>>>>>>>>>> [hotfix][connectors] Improve JavaDocs for
>>>> > > >> > > >>>>>>>>>>>>>>> SingleThreadFetcherManager
>>>> > > >> > > >>>>>>>>>>>>>>>>>>>
>>>> > > >> > > >>>>>>>>>>>>>>>>>>> commit
>>>> > > >> > > >>> 4700bb5dde3303cbe98882f6beb7379425717b01
>>>> > > >> > > >>>>>>>>>>>>>>>>>>> [FLINK-19225][connectors] Various small
>>>> > > >> > > >>>>> improvements
>>>> > > >> > > >>>>>>> to
>>>> > > >> > > >>>>>>>>>>>>>>>>> SourceReaderBase
>>>> > > >> > > >>>>>>>>>>>>>>>>>>> (part 2)
>>>> > > >> > > >>>>>>>>>>>>>>>>>>>
>>>> > > >> > > >>>>>>>>>>>>>>>>>>> commit
>>>> > > >> > > >>> 12261c6b7ed6478a9b9f6a69cb58246b83cab9b7
>>>> > > >> > > >>>>>>>>>>>>>>>>>>> [FLINK-17393][connectors] (follow-up)
>>>> Wakeup
>>>> > > >> > > >>> the
>>>> > > >> > > >>>>>>>>>>> SplitFetchers
>>>> > > >> > > >>>>>>>>>>>>> more
>>>> > > >> > > >>>>>>>>>>>>>>>>>>> elegantly.
>>>> > > >> > > >>>>>>>>>>>>>>>>>>>
>>>> > > >> > > >>>>>>>>>>>>>>>>>>> commit
>>>> > > >> > > >>> c60aaff0249bfd6b5871b7f82e03efc487a54d6b
>>>> > > >> > > >>>>>>>>>>>>>>>>>>> [hotfix][tests] Extend test coverage for
>>>> > > >> > > >>>>>>>>>>>>>>> FutureCompletingBlockingQueue
>>>> > > >> > > >>>>>>>>>>>>>>>>>>>
>>>> > > >> > > >>>>>>>>>>>>>>>>>>> commit
>>>> > > >> > > >>> cef8a587d7fd2fe64cc644da5ed095d82e46f631
>>>> > > >> > > >>>>>>>>>>>>>>>>>>> [FLINK-19245][connectors] Set default
>>>> > > >> > > >> capacity
>>>> > > >> > > >>>> for
>>>> > > >> > > >>>>>>>>>>>>>>>>>>> FutureCompletingBlockingQueue.
>>>> > > >> > > >>>>>>>>>>>>>>>>>>>
>>>> > > >> > > >>>>>>>>>>>>>>>>>>> commit
>>>> > > >> > > >>> 4ea95782b4c6a2538153d4d16ad3f4839c7de0fb
>>>> > > >> > > >>>>>>>>>>>>>>>>>>> [FLINK-19223][connectors] Simplify
>>>> > > >> > > >>> Availability
>>>> > > >> > > >>>>>> Future
>>>> > > >> > > >>>>>>>>>> Model
>>>> > > >> > > >>>>>>>>>>> in
>>>> > > >> > > >>>>>>>>>>>>>>> Base
>>>> > > >> > > >>>>>>>>>>>>>>>>>>> Connector
>>>> > > >> > > >>>>>>>>>>>>>>>>>>>
>>>> > > >> > > >>>>>>>>>>>>>>>>>>> commit
>>>> > > >> > > >>> 511857049ba30c8ff0ee56da551fa4a479dc583e
>>>> > > >> > > >>>>>>>>>>>>>>>>>>> [FLINK-18128][connectors] Ensure idle
>>>> split
>>>> > > >> > > >>>>> fetchers
>>>> > > >> > > >>>>>>>>> lead
>>>> > > >> > > >>>>>>>>>> to
>>>> > > >> > > >>>>>>>>>>>>>>>>> availability
>>>> > > >> > > >>>>>>>>>>>>>>>>>>> notifications.
>>>> > > >> > > >>>>>>>>>>>>>>>>>>>
>>>> > > >> > > >>>>>>>>>>>>>>>>>>> commit
>>>> > > >> > > >>> a8206467af0830dcb89623ea068b5ca3b3450c92
>>>> > > >> > > >>>>>>>>>>>>>>>>>>> [refactor][core] Eagerly initialize the
>>>> > > >> > > >>>> FetchTask
>>>> > > >> > > >>>>> to
>>>> > > >> > > >>>>>>>>>> support
>>>> > > >> > > >>>>>>>>>>>>> proper
>>>> > > >> > > >>>>>>>>>>>>>>>>> unit
>>>> > > >> > > >>>>>>>>>>>>>>>>>>> testing
>>>> > > >> > > >>>>>>>>>>>>>>>>>>>
>>>> > > >> > > >>>>>>>>>>>>>>>>>>> commit
>>>> > > >> > > >>> 3b2f54bcb437f98e6137c904045cc51072b5c06b
>>>> > > >> > > >>>>>>>>>>>>>>>>>>> [hotfix][tests] Move constants in
>>>> > > >> > > >>>> SplitFetcherTest
>>>> > > >> > > >>>>>>>>> relevant
>>>> > > >> > > >>>>>>>>>>> to
>>>> > > >> > > >>>>>>>>>>>>> only
>>>> > > >> > > >>>>>>>>>>>>>>>>> one
>>>> > > >> > > >>>>>>>>>>>>>>>>>>> test into test method
>>>> > > >> > > >>>>>>>>>>>>>>>>>>>
>>>> > > >> > > >>>>>>>>>>>>>>>>>>> commit
>>>> > > >> > > >>> d7625760a75a508bf05bcddc380bb4d62ee1743e
>>>> > > >> > > >>>>>>>>>>>>>>>>>>> [FLINK-19225][connectors] Various small
>>>> > > >> > > >>>>> improvements
>>>> > > >> > > >>>>>>> to
>>>> > > >> > > >>>>>>>>>>>>>>>>> SourceReaderBase
>>>> > > >> > > >>>>>>>>>>>>>>>>>>>
>>>> > > >> > > >>>>>>>>>>>>>>>>>>> commit
>>>> > > >> > > >>> a5b0d3297748c1be47ad579a88f24df2255a8df1
>>>> > > >> > > >>>>>>>>>>>>>>>>>>> [FLINK-17393][connectors] Wakeup the
>>>> > > >> > > >>>> SplitFetchers
>>>> > > >> > > >>>>>>> more
>>>> > > >> > > >>>>>>>>>>>>> elegantly.
>>>> > > >> > > >>>>>>>>>>>>>>>>>>>
>>>> > > >> > > >>>>>>>>>>>>>>>>>>> commit
>>>> > > >> > > >>> f42a3ebc3e81a034b7221a803c153636fef34903
>>>> > > >> > > >>>>>>>>>>>>>>>>>>> [FLINK-18680][connectors] Make connector
>>>> > > >> > > >> base
>>>> > > >> > > >>>>>>>>>>>> RecordsWithSplitIds
>>>> > > >> > > >>>>>>>>>>>>>>> more
>>>> > > >> > > >>>>>>>>>>>>>>>>>>> lightweight.
>>>> > > >> > > >>>>>>>>>>>>>>>>>>>
>>>> > > >> > > >>>>>>>>>>>>>>>>>>> commit
>>>> > > >> > > >>> e3d273de822b085183d09b275a445879ff94b350
>>>> > > >> > > >>>>>>>>>>>>>>>>>>> [FLINK-19162][connectors] Add
>>>> 'recycle()' to
>>>> > > >> > > >>> the
>>>> > > >> > > >>>>>>>>>>>>>>> RecordsWithSplitIds
>>>> > > >> > > >>>>>>>>>>>>>>>>> to
>>>> > > >> > > >>>>>>>>>>>>>>>>>>> support reuse of heavy objects.
>>>> > > >> > > >>>>>>>>>>>>>>>>>>>
>>>> > > >> > > >>>>>>>>>>>>>>>>>>> commit
>>>> > > >> > > >>> 8ebc464c2520453a70001cd712abc8dee6ee89e0
>>>> > > >> > > >>>>>>>>>>>>>>>>>>> [hotfix][testing] Add a set of
>>>> > > >> > > >> parameterizable
>>>> > > >> > > >>>>>> testing
>>>> > > >> > > >>>>>>>>>> mocks
>>>> > > >> > > >>>>>>>>>>>> for
>>>> > > >> > > >>>>>>>>>>>>>>> the
>>>> > > >> > > >>>>>>>>>>>>>>>>>> Split
>>>> > > >> > > >>>>>>>>>>>>>>>>>>> Reader API
>>>> > > >> > > >>>>>>>>>>>>>>>>>>>
>>>> > > >> > > >>>>>>>>>>>>>>>>>>> commit
>>>> > > >> > > >>> 930a07438be1185388d7150640f294dfe2a5d378
>>>> > > >> > > >>>>>>>>>>>>>>>>>>> [hotfix][connectors] Add
>>>> RequestSplitEvent
>>>> > > >> > > >> to
>>>> > > >> > > >>>>>>>>>>>>>>> 'flink-connector-base'
>>>> > > >> > > >>>>>>>>>>>>>>>>>>>
>>>> > > >> > > >>>>>>>>>>>>>>>>>>> commit
>>>> > > >> > > >>> e945ce8a933bc378844782f784ca473c767ca159
>>>> > > >> > > >>>>>>>>>>>>>>>>>>> [refactor][connectors] Backport of the
>>>> > > >> > > >>>>>> connector-base
>>>> > > >> > > >>>>>>>>>>> exception
>>>> > > >> > > >>>>>>>>>>>>>>>>> handling
>>>> > > >> > > >>>>>>>>>>>>>>>>>>> from the Kafka Connector Pull Request
>>>> > > >> > > >>>>>>>>>>>>>>>>>>>
>>>> > > >> > > >>>>>>>>>>>>>>>>>>>
>>>> > > >> > > >>>>>>>>>>>>>>>>>>> On Wed, Oct 28, 2020 at 5:49 PM Steven
>>>> Wu <
>>>> > > >> > > >>>>>>>>>>>> stevenz3wu@gmail.com>
>>>> > > >> > > >>>>>>>>>>>>>>>>> wrote:
>>>> > > >> > > >>>>>>>>>>>>>>>>>>>
>>>> > > >> > > >>>>>>>>>>>>>>>>>>>> I would love to see this FLIP-27 source
>>>> > > >> > > >>>>> interface
>>>> > > >> > > >>>>>>>>>>> improvement
>>>> > > >> > > >>>>>>>>>>>>> [1]
>>>> > > >> > > >>>>>>>>>>>>>>>>> made
>>>> > > >> > > >>>>>>>>>>>>>>>>>> to
>>>> > > >> > > >>>>>>>>>>>>>>>>>>>> 1.11.3.
>>>> > > >> > > >>>>>>>>>>>>>>>>>>>>
>>>> > > >> > > >>>>>>>>>>>>>>>>>>>> [1]
>>>> > > >> > > >>>>>>> https://issues.apache.org/jira/browse/FLINK-19698
>>>> > > >> > > >>>>>>>>>>>>>>>>>>>>
>>>> > > >> > > >>>>>>>>>>>>>>>>>>>> On Wed, Oct 28, 2020 at 12:32 AM Tzu-Li
>>>> > > >> > > >>>> (Gordon)
>>>> > > >> > > >>>>>>> Tai <
>>>> > > >> > > >>>>>>>>>>>>>>>>>>> tzulitai@apache.org>
>>>> > > >> > > >>>>>>>>>>>>>>>>>>>> wrote:
>>>> > > >> > > >>>>>>>>>>>>>>>>>>>>
>>>> > > >> > > >>>>>>>>>>>>>>>>>>>>> Thanks for the replies so far!
>>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>
>>>> > > >> > > >>>>>>>>>>>>>>>>>>>>> Just to provide a brief update on the
>>>> > > >> > > >>> status
>>>> > > >> > > >>>>> of
>>>> > > >> > > >>>>>>>>>> blockers
>>>> > > >> > > >>>>>>>>>>>> for
>>>> > > >> > > >>>>>>>>>>>>>>>>> 1.11.3
>>>> > > >> > > >>>>>>>>>>>>>>>>>> so
>>>> > > >> > > >>>>>>>>>>>>>>>>>>>> far:
>>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>
>>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>
>>>> > > >> > > >>>>>>>>>>>>>>>>>>>>> *PR opened, pending reviewer*-
>>>> > > >> > > >>> [FLINK-19717]
>>>> > > >> > > >>>>>>>>>>>>>>>>>> SourceReaderBase.pollNext
>>>> > > >> > > >>>>>>>>>>>>>>>>>>>> may
>>>> > > >> > > >>>>>>>>>>>>>>>>>>>>> return END_OF_INPUT if
>>>> SplitReader.fetch
>>>> > > >> > > >>>>> throws
>>>> > > >> > > >>>>>> (
>>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>
>>>> > > >> > > >>> https://github.com/apache/flink/pull/13776)
>>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>
>>>> > > >> > > >>>>>>>>>>>>>>>>>>>>> *PR opened, reviewed + close to being
>>>> > > >> > > >>>> merged*
>>>> > > >> > > >>>>>>>>>>>>>>>>>>>>> - [FLINK-19741] Timer service should
>>>> > > >> > > >> skip
>>>> > > >> > > >>>>>>> restoring
>>>> > > >> > > >>>>>>>>>> from
>>>> > > >> > > >>>>>>>>>>>> raw
>>>> > > >> > > >>>>>>>>>>>>>>> keyed
>>>> > > >> > > >>>>>>>>>>>>>>>>>>> stream
>>>> > > >> > > >>>>>>>>>>>>>>>>>>>>> if it isn't the writer (
>>>> > > >> > > >>>>>>>>>>>>>>>>> https://github.com/apache/flink/pull/13761
>>>> )
>>>> > > >> > > >>>>>>>>>>>>>>>>>>>>> - [FLINK-19748] Raw keyed stream key
>>>> > > >> > > >> group
>>>> > > >> > > >>>>>>> iterator
>>>> > > >> > > >>>>>>>>>>> should
>>>> > > >> > > >>>>>>>>>>>> be
>>>> > > >> > > >>>>>>>>>>>>>>>>>> skipping
>>>> > > >> > > >>>>>>>>>>>>>>>>>>>>> unwritten key groups (
>>>> > > >> > > >>>>>>>>>>>>>>> https://github.com/apache/flink/pull/13772)
>>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>
>>>> > > >> > > >>>>>>>>>>>>>>>>>>>>> *Merged*
>>>> > > >> > > >>>>>>>>>>>>>>>>>>>>> - [FLINK-19154] Application mode
>>>> deletes
>>>> > > >> > > >>> HA
>>>> > > >> > > >>>>> data
>>>> > > >> > > >>>>>>> in
>>>> > > >> > > >>>>>>>>>> case
>>>> > > >> > > >>>>>>>>>>> of
>>>> > > >> > > >>>>>>>>>>>>>>>>> suspended
>>>> > > >> > > >>>>>>>>>>>>>>>>>>>>> ZooKeeper connection
>>>> > > >> > > >>>>>>>>>>>>>>>>>>>>> - [FLINK-19569] Upgrade ICU4J to 67.1+
>>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>
>>>> > > >> > > >>>>>>>>>>>>>>>>>>>>> Right now as it seems, progress is
>>>> > > >> > > >> mainly
>>>> > > >> > > >>>>>> blocked
>>>> > > >> > > >>>>>>>>> on a
>>>> > > >> > > >>>>>>>>>>>>> reviewer
>>>> > > >> > > >>>>>>>>>>>>>>>>> for
>>>> > > >> > > >>>>>>>>>>>>>>>>>>>>> FLINK-19717.
>>>> > > >> > > >>>>>>>>>>>>>>>>>>>>> Meanwhile, Xintong is keeping an eye on
>>>> > > >> > > >>> test
>>>> > > >> > > >>>>>>>>>>> instabilities
>>>> > > >> > > >>>>>>>>>>>>> [1]
>>>> > > >> > > >>>>>>>>>>>>>>> to
>>>> > > >> > > >>>>>>>>>>>>>>>>> see
>>>> > > >> > > >>>>>>>>>>>>>>>>>>> if
>>>> > > >> > > >>>>>>>>>>>>>>>>>>>>> there are any fixes that should be
>>>> > > >> > > >> applied
>>>> > > >> > > >>>> to
>>>> > > >> > > >>>>>>>>>>>> `release-1.11`.
>>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>
>>>> > > >> > > >>>>>>>>>>>>>>>>>>>>> This is also a reminder, that if there
>>>> > > >> > > >> are
>>>> > > >> > > >>>>> other
>>>> > > >> > > >>>>>>>>>> blockers
>>>> > > >> > > >>>>>>>>>>>>> that
>>>> > > >> > > >>>>>>>>>>>>>>> we
>>>> > > >> > > >>>>>>>>>>>>>>>>>> need
>>>> > > >> > > >>>>>>>>>>>>>>>>>>> to
>>>> > > >> > > >>>>>>>>>>>>>>>>>>>>> be aware of, or a need to re-establish
>>>> > > >> > > >>>>> estimated
>>>> > > >> > > >>>>>>>>> time
>>>> > > >> > > >>>>>>>>>> for
>>>> > > >> > > >>>>>>>>>>>>>>> getting
>>>> > > >> > > >>>>>>>>>>>>>>>>>> fixes
>>>> > > >> > > >>>>>>>>>>>>>>>>>>>> in
>>>> > > >> > > >>>>>>>>>>>>>>>>>>>>> and delay the RC for 1.11.3, please do
>>>> > > >> > > >> let
>>>> > > >> > > >>>> us
>>>> > > >> > > >>>>>>> know!
>>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>
>>>> > > >> > > >>>>>>>>>>>>>>>>>>>>> Cheers,
>>>> > > >> > > >>>>>>>>>>>>>>>>>>>>> Gordon
>>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>
>>>> > > >> > > >>>>>>>>>>>>>>>>>>>>> [1]
>>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>
>>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>
>>>> > > >> > > >>>>>>>>>>>>>>>>>>>>
>>>> > > >> > > >>>>>>>>>>>>>>>>>>>
>>>> > > >> > > >>>>>>>>>>>>>>>>>>
>>>> > > >> > > >>>>>>>>>>>>>>>>>
>>>> > > >> > > >>>>>>>>>>>>>>>
>>>> > > >> > > >>>>>>>>>>>>>
>>>> > > >> > > >>>>>>>>>>>>
>>>> > > >> > > >>>>>>>>>>>
>>>> > > >> > > >>>>>>>>>>
>>>> > > >> > > >>>>>>>>>
>>>> > > >> > > >>>>>>>
>>>> > > >> > > >>>>>>
>>>> > > >> > > >>>>>
>>>> > > >> > > >>>>
>>>> > > >> > > >>>
>>>> > > >> > > >>
>>>> > > >> > >
>>>> > > >> >
>>>> > > >>
>>>> > >
>>>> >
>>>> https://issues.apache.org/jira/issues/?jql=project%20%3D%20FLINK%20AND%20status%20in%20(Open%2C%20%22In%20Progress%22%2C%20Reopened)%20AND%20affectedVersion%20in%20(1.11.0%2C%201.11.1%2C%201.11.2)%20AND%20labels%20%3D%20test-stability
>>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>
>>>> > > >> > > >>>>>>>>>>>>>>>>>>>>> On Mon, Oct 26, 2020 at 9:43 PM Kostas
>>>> > > >> > > >>>>> Kloudas <
>>>> > > >> > > >>>>>>>>>>>>>>>>> kkloudas@apache.org>
>>>> > > >> > > >>>>>>>>>>>>>>>>>>>>> wrote:
>>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>
>>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>> +1 for releasing Flink 1.11.3 as it
>>>> > > >> > > >>>>> contains a
>>>> > > >> > > >>>>>>>>> number
>>>> > > >> > > >>>>>>>>>>> of
>>>> > > >> > > >>>>>>>>>>>>>>>>> important
>>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>> fixes and thanks Gordon and Xintong
>>>> > > >> > > >> for
>>>> > > >> > > >>>>>>>>> volunteering.
>>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>
>>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>> Cheers,
>>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>> Kostas
>>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>
>>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>> On Mon, Oct 26, 2020 at 4:37 AM Yu Li
>>>> > > >> > > >> <
>>>> > > >> > > >>>>>>>>>>> carp84@gmail.com>
>>>> > > >> > > >>>>>>>>>>>>>>> wrote:
>>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>
>>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>> +1 for releasing Flink 1.11.3, and
>>>> > > >> > > >>>> thanks
>>>> > > >> > > >>>>>>> Gordon
>>>> > > >> > > >>>>>>>>>> and
>>>> > > >> > > >>>>>>>>>>>>>>> Xintong
>>>> > > >> > > >>>>>>>>>>>>>>>>> for
>>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>> volunteering as our release
>>>> > > >> > > >> managers.
>>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>
>>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>> Best Regards,
>>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>> Yu
>>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>
>>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>
>>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>> On Mon, 26 Oct 2020 at 09:45,
>>>> > > >> > > >> Xintong
>>>> > > >> > > >>>>> Song <
>>>> > > >> > > >>>>>>>>>>>>>>>>>> tonysong820@gmail.com>
>>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>> wrote:
>>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>
>>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>> Thanks Gordan for starting this
>>>> > > >> > > >>>>>> discussion.
>>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>> My pleasure to help with the
>>>> > > >> > > >> release
>>>> > > >> > > >>>>>>> process.
>>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>
>>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>> Thank you~
>>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>
>>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>> Xintong Song
>>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>
>>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>
>>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>
>>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>> On Fri, Oct 23, 2020 at 11:29 PM
>>>> > > >> > > >>> Till
>>>> > > >> > > >>>>>>>>> Rohrmann <
>>>> > > >> > > >>>>>>>>>>>>>>>>>>>> trohrmann@apache.org
>>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>
>>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>
>>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>> Thanks for starting this
>>>> > > >> > > >>> discussion
>>>> > > >> > > >>>>>>> Gordon.
>>>> > > >> > > >>>>>>>>>> There
>>>> > > >> > > >>>>>>>>>>>> are
>>>> > > >> > > >>>>>>>>>>>>>>> over
>>>> > > >> > > >>>>>>>>>>>>>>>>>> 100
>>>> > > >> > > >>>>>>>>>>>>>>>>>>>>> issues
>>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>> which are fixed for 1.11.3.
>>>> > > >> > > >> Hence
>>>> > > >> > > >>> +1
>>>> > > >> > > >>>>>> for a
>>>> > > >> > > >>>>>>>>>>> soonish
>>>> > > >> > > >>>>>>>>>>>>>>> 1.11.3
>>>> > > >> > > >>>>>>>>>>>>>>>>>>>> release.
>>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>> Thanks
>>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>> for volunteering as our release
>>>> > > >> > > >>>>> managers
>>>> > > >> > > >>>>>>>>> Gordon
>>>> > > >> > > >>>>>>>>>>> and
>>>> > > >> > > >>>>>>>>>>>>>>>>> Xintong!
>>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>
>>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>> Cheers,
>>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>> Till
>>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>
>>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>> On Fri, Oct 23, 2020 at 5:02 PM
>>>> > > >> > > >>>> Tzu-Li
>>>> > > >> > > >>>>>>>>> (Gordon)
>>>> > > >> > > >>>>>>>>>>>> Tai <
>>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>> tzulitai@apache.org
>>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>
>>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>
>>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>> Hi,
>>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>>
>>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>> Xintong and I would like to
>>>> > > >> > > >>> start a
>>>> > > >> > > >>>>>>>>> discussion
>>>> > > >> > > >>>>>>>>>>> for
>>>> > > >> > > >>>>>>>>>>>>>>>>> releasing
>>>> > > >> > > >>>>>>>>>>>>>>>>>>>> Flink
>>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>> 1.11.3
>>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>> soon.
>>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>>
>>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>> It seems like we already have a
>>>> > > >> > > >>> few
>>>> > > >> > > >>>>>>>>> pressing
>>>> > > >> > > >>>>>>>>>>>> issues
>>>> > > >> > > >>>>>>>>>>>>>>> that
>>>> > > >> > > >>>>>>>>>>>>>>>>>> needs
>>>> > > >> > > >>>>>>>>>>>>>>>>>>>> to
>>>> > > >> > > >>>>>>>>>>>>>>>>>>>>> be
>>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>> included in a new hotfix
>>>> > > >> > > >> release:
>>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>>
>>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>>   - Heap-based timers’ restore
>>>> > > >> > > >>>>>> behaviour
>>>> > > >> > > >>>>>>>>> is
>>>> > > >> > > >>>>>>>>>>>>> causing a
>>>> > > >> > > >>>>>>>>>>>>>>>>>>> critical
>>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>> recovery
>>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>>   issue for StateFun [1] [2]
>>>> > > >> > > >>> [3].
>>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>>   - There are several
>>>> > > >> > > >> robustness
>>>> > > >> > > >>>>>> issues
>>>> > > >> > > >>>>>>>>> for
>>>> > > >> > > >>>>>>>>>> the
>>>> > > >> > > >>>>>>>>>>>>>>> FLIP-27
>>>> > > >> > > >>>>>>>>>>>>>>>>> new
>>>> > > >> > > >>>>>>>>>>>>>>>>>>>>> source
>>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>> API,
>>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>>   such as [4]. We already have
>>>> > > >> > > >>>> some
>>>> > > >> > > >>>>>>> users
>>>> > > >> > > >>>>>>>>>> using
>>>> > > >> > > >>>>>>>>>>>> the
>>>> > > >> > > >>>>>>>>>>>>>>>>> FLIP-27
>>>> > > >> > > >>>>>>>>>>>>>>>>>>> API
>>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>> with
>>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>> 1.11.x,
>>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>>   so it would be important to
>>>> > > >> > > >>> get
>>>> > > >> > > >>>>>> those
>>>> > > >> > > >>>>>>>>> fixes
>>>> > > >> > > >>>>>>>>>>> in
>>>> > > >> > > >>>>>>>>>>>>> for
>>>> > > >> > > >>>>>>>>>>>>>>>>> 1.11.x
>>>> > > >> > > >>>>>>>>>>>>>>>>>>> as
>>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>> well.
>>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>>
>>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>> Apart from the issues that are
>>>> > > >> > > >>>>> already
>>>> > > >> > > >>>>>>>>> marked
>>>> > > >> > > >>>>>>>>>> as
>>>> > > >> > > >>>>>>>>>>>>>>> blocker
>>>> > > >> > > >>>>>>>>>>>>>>>>> for
>>>> > > >> > > >>>>>>>>>>>>>>>>>>>>> 1.11.3
>>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>> in
>>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>> our
>>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>> JIRA [5], please let us know in
>>>> > > >> > > >>>> this
>>>> > > >> > > >>>>>>>>> thread if
>>>> > > >> > > >>>>>>>>>>>> there
>>>> > > >> > > >>>>>>>>>>>>>>> is
>>>> > > >> > > >>>>>>>>>>>>>>>>>>> already
>>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>> ongoing
>>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>> work for other important fixes
>>>> > > >> > > >>> that
>>>> > > >> > > >>>>> we
>>>> > > >> > > >>>>>>>>> should
>>>> > > >> > > >>>>>>>>>>> try
>>>> > > >> > > >>>>>>>>>>>> to
>>>> > > >> > > >>>>>>>>>>>>>>>>>> include.
>>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>>
>>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>> Xintong and I would like to
>>>> > > >> > > >>>> volunteer
>>>> > > >> > > >>>>>> for
>>>> > > >> > > >>>>>>>>>>> managing
>>>> > > >> > > >>>>>>>>>>>>>>> this
>>>> > > >> > > >>>>>>>>>>>>>>>>>>> release,
>>>> > > >> > > >>>>>>>>>>>>>>>>>>>>> and
>>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>> will
>>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>> try to communicate the priority
>>>> > > >> > > >>> of
>>>> > > >> > > >>>>>>> pending
>>>> > > >> > > >>>>>>>>>>>> blockers
>>>> > > >> > > >>>>>>>>>>>>>>> over
>>>> > > >> > > >>>>>>>>>>>>>>>>> the
>>>> > > >> > > >>>>>>>>>>>>>>>>>>>> next
>>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>> few
>>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>> days.
>>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>> Since the aforementioned issues
>>>> > > >> > > >>> are
>>>> > > >> > > >>>>>> quite
>>>> > > >> > > >>>>>>>>>>>> critical,
>>>> > > >> > > >>>>>>>>>>>>>>> we’d
>>>> > > >> > > >>>>>>>>>>>>>>>>>> like
>>>> > > >> > > >>>>>>>>>>>>>>>>>>> to
>>>> > > >> > > >>>>>>>>>>>>>>>>>>>>> aim
>>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>> for a *feature
>>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>> freeze by the end of next week
>>>> > > >> > > >>>> (Oct.
>>>> > > >> > > >>>>>>> 30th)*
>>>> > > >> > > >>>>>>>>>> and
>>>> > > >> > > >>>>>>>>>>>>> start
>>>> > > >> > > >>>>>>>>>>>>>>> the
>>>> > > >> > > >>>>>>>>>>>>>>>>>>>> release
>>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>> voting
>>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>> process the week after.
>>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>> If that is too short of a
>>>> > > >> > > >> notice
>>>> > > >> > > >>>> and
>>>> > > >> > > >>>>>> you
>>>> > > >> > > >>>>>>>>> might
>>>> > > >> > > >>>>>>>>>>>> need
>>>> > > >> > > >>>>>>>>>>>>>>> more
>>>> > > >> > > >>>>>>>>>>>>>>>>>> time,
>>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>> please
>>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>> let
>>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>> us know!
>>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>>
>>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>> Cheers,
>>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>> Gordon
>>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>>
>>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>> [1]
>>>> > > >> > > >>>>>>>>>>>>>
>>>> https://issues.apache.org/jira/browse/FLINK-19692
>>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>> [2]
>>>> > > >> > > >>>>>>>>>>>>>
>>>> https://issues.apache.org/jira/browse/FLINK-19741
>>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>> [3]
>>>> > > >> > > >>>>>>>>>>>>>
>>>> https://issues.apache.org/jira/browse/FLINK-19748
>>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>> [4]
>>>> > > >> > > >>>>>>>>>>>>>
>>>> https://issues.apache.org/jira/browse/FLINK-19717
>>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>> [5]
>>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>>
>>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>>
>>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>
>>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>
>>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>
>>>> > > >> > > >>>>>>>>>>>>>>>>>>>>
>>>> > > >> > > >>>>>>>>>>>>>>>>>>>
>>>> > > >> > > >>>>>>>>>>>>>>>>>>
>>>> > > >> > > >>>>>>>>>>>>>>>>>
>>>> > > >> > > >>>>>>>>>>>>>>>
>>>> > > >> > > >>>>>>>>>>>>>
>>>> > > >> > > >>>>>>>>>>>>
>>>> > > >> > > >>>>>>>>>>>
>>>> > > >> > > >>>>>>>>>>
>>>> > > >> > > >>>>>>>>>
>>>> > > >> > > >>>>>>>
>>>> > > >> > > >>>>>>
>>>> > > >> > > >>>>>
>>>> > > >> > > >>>>
>>>> > > >> > > >>>
>>>> > > >> > > >>
>>>> > > >> > >
>>>> > > >> >
>>>> > > >>
>>>> > >
>>>> >
>>>> https://issues.apache.org/jira/issues/?jql=project%20%3D%20FLINK%20AND%20priority%20%3D%20Blocker%20AND%20fixVersion%20%3D%201.11.3
>>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>>
>>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>
>>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>
>>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>
>>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>
>>>> > > >> > > >>>>>>>>>>>>>>>>>>>>
>>>> > > >> > > >>>>>>>>>>>>>>>>>>>
>>>> > > >> > > >>>>>>>>>>>>>>>>>>
>>>> > > >> > > >>>>>>>>>>>>>>>>>>
>>>> > > >> > > >>>>>>>>>>>>>>>>>> --
>>>> > > >> > > >>>>>>>>>>>>>>>>>> Best, Jingsong Lee
>>>> > > >> > > >>>>>>>>>>>>>>>>>>
>>>> > > >> > > >>>>>>>>>>>>>>>>>
>>>> > > >> > > >>>>>>>>>>>>>>>>
>>>> > > >> > > >>>>>>>>>>>>>>>
>>>> > > >> > > >>>>>>>>>>>>>>
>>>> > > >> > > >>>>>>>>>>>>>
>>>> > > >> > > >>>>>>>>>>>>
>>>> > > >> > > >>>>>>>>>>>
>>>> > > >> > > >>>>>>>>>>
>>>> > > >> > > >>>>>>>>>
>>>> > > >> > > >>>>>>>>
>>>> > > >> > > >>>>>>>
>>>> > > >> > > >>>>>>
>>>> > > >> > > >>>>>
>>>> > > >> > > >>>>
>>>> > > >> > > >>>
>>>> > > >> > > >>
>>>> > > >> > >
>>>> > > >> > >
>>>> > > >> > >
>>>> > > >> >
>>>> > > >>
>>>> > > >
>>>> > >
>>>> >
>>>>
>>>

Re: [DISCUSS] Releasing Apache Flink 1.11.3

Posted by Xintong Song <to...@gmail.com>.
Updates on the release progress.

- FLIP-27 is stabilizing on the master branch. Becket is working on the
final backportings, which is estimated to be finished this weekend.
- There's a new blocker FLINK-20442, a license problem discovered during
the 1.12.0 RC2 testings, which is already fixed for 1.12.

We are hoping to create a RC next Monday, if things go well for the above
issues.


Thank you~

Xintong Song



On Fri, Nov 27, 2020 at 9:49 AM Xintong Song <to...@gmail.com> wrote:

> Hi devs,
>
> Updates on the release progress.
>
> There are a couple of new blockers emerged since the last update, all of
> which have been resolved in the past week.
>
> The release 1.11.3 is still blocked on FLIP-27.
> Stephan and Becket are trying to first get the feature stabilized on the
> master branch for release 1.12. After that, the remaining changes will be
> backported to 1.11, and the release 1.11.3 will be unblocked.
>
> We are hoping to deliver the first RC by the end of next week. During this
> time, if there's any other things that you believe should be included in
> this release, please reach out to either me or Gordon.
>
> Sorry for the delay.
>
> Thank you~
>
> Xintong Song
>
>
>
> On Mon, Nov 23, 2020 at 2:03 PM Xintong Song <to...@gmail.com>
> wrote:
>
>> Thanks for the notice, Zhu.
>> Let's have the discussion on the jira ticket.
>>
>> Thank you~
>>
>> Xintong Song
>>
>>
>>
>> On Mon, Nov 23, 2020 at 12:10 PM Zhu Zhu <re...@gmail.com> wrote:
>>
>>> Hi Xintong,
>>> We just found an issue FLINK-20285 and I think it is a blocker for
>>> 1.11.3.
>>>
>>> Thanks,
>>> Zhu
>>>
>>> Gyula Fóra <gy...@gmail.com> 于2020年11月18日周三 下午9:36写道：
>>>
>>> > Hi All!
>>> >
>>> > I have found the following issue today which might be considered a
>>> blocker
>>> > for this release as well:
>>> >
>>> > https://issues.apache.org/jira/browse/FLINK-20221
>>> >
>>> > Could someone please quickly provide a second set of eyes and confirm
>>> that
>>> > this is indeed a big problem? :)
>>> >
>>> > Thank you!
>>> > Gyula
>>> >
>>> > On Wed, Nov 18, 2020 at 5:12 AM Xintong Song <to...@gmail.com>
>>> > wrote:
>>> >
>>> > > Hi devs,
>>> > >
>>> > > Updates on the progress of preparing the 1.11.3 release.
>>> > >
>>> > > We are approaching the creation of our first release candidate.
>>> Thanks
>>> > all
>>> > > for the efforts so far.
>>> > >
>>> > > # Ongoing threads
>>> > >
>>> > >    - *FLIP-27 backportings:* 74 out of 102 commits are already
>>> > backported.
>>> > >    Stephan and Becket are actively working on this.
>>> > >    - *Blockers:* There are 2 remaining blockers, both have been
>>> fixed on
>>> > >    the master branch and should be backported along with the FLIP-27
>>> > > changes.
>>> > >
>>> > > As soon as the above threads are done, we will create our first
>>> release
>>> > > candidate. If there's any other issues that you believe should be a
>>> > release
>>> > > blocker, please reach out to either me or Gordon.
>>> > >
>>> > > # JIRA version
>>> > >
>>> > > Version 1.11.4 has been created on JIRA. I've already tagged some of
>>> the
>>> > > tickets with the new fix version. Please feel free to continue with
>>> the
>>> > > efforts and merge once they are ready. I'll double check and update
>>> the
>>> > fix
>>> > > versions before the release.
>>> > >
>>> > > Thank you~
>>> > >
>>> > > Xintong Song
>>> > >
>>> > >
>>> > >
>>> > > On Thu, Nov 12, 2020 at 1:31 PM Xintong Song <to...@gmail.com>
>>> > > wrote:
>>> > >
>>> > > > Thanks for the notice and fix, Roman.
>>> > > >
>>> > > > Thank you~
>>> > > >
>>> > > > Xintong Song
>>> > > >
>>> > > >
>>> > > >
>>> > > > On Wed, Nov 11, 2020 at 5:53 PM Khachatryan Roman <
>>> > > > khachatryan.roman@gmail.com> wrote:
>>> > > >
>>> > > >> Hi,
>>> > > >>
>>> > > >> I'd like FLINK-20079 [1] to be merged into 1.11 and included in
>>> > 1.11.3.
>>> > > >>
>>> > > >> [1] https://issues.apache.org/jira/browse/FLINK-20079
>>> > > >>
>>> > > >> Regards,
>>> > > >> Roman
>>> > > >>
>>> > > >>
>>> > > >> On Tue, Nov 10, 2020 at 8:21 AM Xintong Song <
>>> tonysong820@gmail.com>
>>> > > >> wrote:
>>> > > >>
>>> > > >> > Thanks for the notice, Dian.
>>> > > >> >
>>> > > >> > Thank you~
>>> > > >> >
>>> > > >> > Xintong Song
>>> > > >> >
>>> > > >> >
>>> > > >> >
>>> > > >> > On Tue, Nov 10, 2020 at 10:19 AM Dian Fu <dian0511.fu@gmail.com
>>> >
>>> > > wrote:
>>> > > >> >
>>> > > >> > > Hi Xintong,
>>> > > >> > >
>>> > > >> > > I want to bring one more issue to your attention [1]. The test
>>> > case
>>> > > >> > > UnalignedCheckpointCompatibilityITCase.test failed several
>>> times
>>> > in
>>> > > >> the
>>> > > >> > > last nightly test of release-1.11. We need to figure out if
>>> it's
>>> > > just
>>> > > >> an
>>> > > >> > > instable test or caused by recent changes.
>>> > > >> > >
>>> > > >> > > [1] https://issues.apache.org/jira/browse/FLINK-20065
>>> > > >> > >
>>> > > >> > > > 在 2020年11月10日，上午9:24，Xintong Song <to...@gmail.com>
>>> 写道：
>>> > > >> > > >
>>> > > >> > > > Thanks for the replies.
>>> > > >> > > >
>>> > > >> > > > Thank you~
>>> > > >> > > >
>>> > > >> > > > Xintong Song
>>> > > >> > > >
>>> > > >> > > >
>>> > > >> > > >
>>> > > >> > > > On Tue, Nov 10, 2020 at 1:09 AM Becket Qin <
>>> > becket.qin@gmail.com>
>>> > > >> > wrote:
>>> > > >> > > >
>>> > > >> > > >> Hi Xintong,
>>> > > >> > > >>
>>> > > >> > > >> Thanks for driving the release. Just want to sync up on the
>>> > > FLIP-27
>>> > > >> > > >> backporting. Stephan and I are still trying to backport a
>>> bunch
>>> > > of
>>> > > >> > > patches
>>> > > >> > > >> of Source to 1.11.3. Including:
>>> > > >> > > >>
>>> > > >> > > >> [FLINK-19698][connector/common] Add a close() method to the
>>> > > >> > SplitReader.
>>> > > >> > > >> [FLINK-19717] SourceReaderBase.pollNext may return
>>> END_OF_INPUT
>>> > > if
>>> > > >> > > >> SplitReader.fetch throws
>>> > > >> > > >> [FLINK-19535] [connector/common] Avoid failing a job
>>> multiple
>>> > > >> times in
>>> > > >> > > >> SourceCoordinator.
>>> > > >> > > >> [FLINK-19265] [FLINK-20049][core] Source API final
>>> adjustments.
>>> > > >> > > >>
>>> > > >> > > >> and a few more fixes.
>>> > > >> > > >>
>>> > > >> > > >> We are currently trying to fix them in 1.12 first so it
>>> might
>>> > > take
>>> > > >> a
>>> > > >> > > little
>>> > > >> > > >> longer to backport them to 1.11.3. I think it will probably
>>> > take
>>> > > >> us a
>>> > > >> > > few
>>> > > >> > > >> more days to finish the backport. So that would roughly be
>>> the
>>> > > end
>>> > > >> of
>>> > > >> > > this
>>> > > >> > > >> week.
>>> > > >> > > >>
>>> > > >> > > >> Thanks,
>>> > > >> > > >>
>>> > > >> > > >> Jiangjie (Becket) Qin
>>> > > >> > > >>
>>> > > >> > > >>
>>> > > >> > > >>
>>> > > >> > > >>
>>> > > >> > > >> On Mon, Nov 9, 2020 at 9:57 PM Till Rohrmann <
>>> > > trohrmann@apache.org
>>> > > >> >
>>> > > >> > > wrote:
>>> > > >> > > >>
>>> > > >> > > >>> Yes, I've downgraded FLINK-19816 to critical.
>>> > > >> > > >>>
>>> > > >> > > >>> Cheers,
>>> > > >> > > >>> Till
>>> > > >> > > >>>
>>> > > >> > > >>> On Mon, Nov 9, 2020 at 10:19 AM Xintong Song <
>>> > > >> tonysong820@gmail.com>
>>> > > >> > > >>> wrote:
>>> > > >> > > >>>
>>> > > >> > > >>>> Thanks for the notice, Till.
>>> > > >> > > >>>>
>>> > > >> > > >>>> I just checked and found FLINK-20033 is already fixed.
>>> Shall
>>> > we
>>> > > >> also
>>> > > >> > > >>>> downgrade FLINK-19816 to `Critical`?
>>> > > >> > > >>>>
>>> > > >> > > >>>> Thank you~
>>> > > >> > > >>>>
>>> > > >> > > >>>> Xintong Song
>>> > > >> > > >>>>
>>> > > >> > > >>>>
>>> > > >> > > >>>>
>>> > > >> > > >>>> On Mon, Nov 9, 2020 at 4:42 PM Till Rohrmann <
>>> > > >> trohrmann@apache.org>
>>> > > >> > > >>> wrote:
>>> > > >> > > >>>>
>>> > > >> > > >>>>> I would like to bring one more critical issue to your
>>> > > attention
>>> > > >> > which
>>> > > >> > > >>> is
>>> > > >> > > >>>>> FLINK-20033 [1]. I believe that this issue is actually
>>> > causing
>>> > > >> what
>>> > > >> > > >> has
>>> > > >> > > >>>>> been reported in FLINK-19816 [2]. I hope to have it
>>> fixed by
>>> > > the
>>> > > >> > end
>>> > > >> > > >> of
>>> > > >> > > >>>>> today. Once FLINK-20033 is fixed, I think that we don't
>>> have
>>> > > to
>>> > > >> > block
>>> > > >> > > >>> the
>>> > > >> > > >>>>> release on FLINK-19816.
>>> > > >> > > >>>>>
>>> > > >> > > >>>>> [1] https://issues.apache.org/jira/browse/FLINK-20033
>>> > > >> > > >>>>> [2] https://issues.apache.org/jira/browse/FLINK-19816
>>> > > >> > > >>>>>
>>> > > >> > > >>>>> Cheers,
>>> > > >> > > >>>>> Till
>>> > > >> > > >>>>>
>>> > > >> > > >>>>> On Mon, Nov 9, 2020 at 4:05 AM Xintong Song <
>>> > > >> tonysong820@gmail.com
>>> > > >> > >
>>> > > >> > > >>>> wrote:
>>> > > >> > > >>>>>
>>> > > >> > > >>>>>> Hi devs,
>>> > > >> > > >>>>>>
>>> > > >> > > >>>>>> I'd like to provide an update on the progress of
>>> preparing
>>> > > >> release
>>> > > >> > > >>>>> 1.11.3.
>>> > > >> > > >>>>>>
>>> > > >> > > >>>>>> *Blockers*
>>> > > >> > > >>>>>> We currently have 3 remaining blockers. (3 resolved
>>> and 1
>>> > > >> emerged
>>> > > >> > > >>>>> compared
>>> > > >> > > >>>>>> to last week)
>>> > > >> > > >>>>>>
>>> > > >> > > >>>>>> - [FLINK-19698] Add close() method and
>>> > onCheckpointComplete()
>>> > > >> to
>>> > > >> > > >> the
>>> > > >> > > >>>>>> Source.
>>> > > >> > > >>>>>> The issue has been fixed on the master branch. It's
>>> > currently
>>> > > >> > > >> blocked
>>> > > >> > > >>>> on
>>> > > >> > > >>>>>> the FLIP-27 backportings to backport it to the 1.11
>>> branch.
>>> > > >> > > >>>>>>
>>> > > >> > > >>>>>> - [FLINK-19717] SourceReaderBase.pollNext may return
>>> > > >> END_OF_INPUT
>>> > > >> > > >> if
>>> > > >> > > >>>>>> SplitReader.fetch throws
>>> > > >> > > >>>>>> A PR has been opened and reviewed. From the
>>> discussions on
>>> > > the
>>> > > >> PR,
>>> > > >> > > >> it
>>> > > >> > > >>>>> looks
>>> > > >> > > >>>>>> close to mergeable.
>>> > > >> > > >>>>>>
>>> > > >> > > >>>>>> - [FLINK-19816] Flink restored from a wrong checkpoint
>>> (a
>>> > > very
>>> > > >> old
>>> > > >> > > >>> one
>>> > > >> > > >>>>> and
>>> > > >> > > >>>>>> not the last completed one)
>>> > > >> > > >>>>>> This is a newly emerged blocker and Matthias is
>>> working on
>>> > > it.
>>> > > >> > > >>>>>>
>>> > > >> > > >>>>>> *Test Instabilities*
>>> > > >> > > >>>>>> We currently have 27 test instabilities[1].
>>> > > >> > > >>>>>> AFAIK, none of them are as serious as to block the
>>> 1.11.3
>>> > > >> release.
>>> > > >> > > >>>>>>
>>> > > >> > > >>>>>> *FLIP-27 Backprotings*
>>> > > >> > > >>>>>>
>>> > > >> > > >>>>>> I noticed that there's no jira issues opened on the
>>> FLIP-27
>>> > > >> > > >>> backporting
>>> > > >> > > >>>>>> efforts, which is part of the major efforts planned
>>> for the
>>> > > >> 1.11.3
>>> > > >> > > >>>>> release,
>>> > > >> > > >>>>>> making it hard to track the progress.
>>> > > >> > > >>>>>>
>>> > > >> > > >>>>>>
>>> > > >> > > >>>>>> @Stephan and @Becket, could you please share the
>>> updates on
>>> > > the
>>> > > >> > > >>>>> backporting
>>> > > >> > > >>>>>> efforts? How is the progress and when are the efforts
>>> > > expected
>>> > > >> to
>>> > > >> > > >> be
>>> > > >> > > >>>>>> finished? It would be appreciated and helpful if we can
>>> > have
>>> > > a
>>> > > >> > jira
>>> > > >> > > >>>>> ticket
>>> > > >> > > >>>>>> to track the backporing progress publicly.
>>> > > >> > > >>>>>>
>>> > > >> > > >>>>>>
>>> > > >> > > >>>>>> Thank you~
>>> > > >> > > >>>>>>
>>> > > >> > > >>>>>> Xintong Song
>>> > > >> > > >>>>>>
>>> > > >> > > >>>>>>
>>> > > >> > > >>>>>> [1]
>>> > > >> > > >>>>>>
>>> > > >> > > >>>>>>
>>> > > >> > > >>>>>
>>> > > >> > > >>>>
>>> > > >> > > >>>
>>> > > >> > > >>
>>> > > >> > >
>>> > > >> >
>>> > > >>
>>> > >
>>> >
>>> https://issues.apache.org/jira/browse/FLINK-16908?filter=12348580&jql=project%20%3D%20FLINK%20AND%20resolution%20%3D%20Unresolved%20AND%20affectedVersion%20in%20(1.11.0%2C%201.11.1%2C%201.11.2%2C%201.11.3)%20AND%20labels%20%3D%20test-stability%20ORDER%20BY%20updated%20DESC%2C%20created%20DESC
>>> > > >> > > >>>>>>
>>> > > >> > > >>>>>> On Wed, Nov 4, 2020 at 3:58 PM Tzu-Li (Gordon) Tai <
>>> > > >> > > >>>> tzulitai@apache.org>
>>> > > >> > > >>>>>> wrote:
>>> > > >> > > >>>>>>
>>> > > >> > > >>>>>>> Thanks for the efforts so far with pushing for 1.11.3.
>>> > > >> > > >>>>>>>
>>> > > >> > > >>>>>>> I'd like to provide a mid-week update on how we're
>>> looking
>>> > > >> with
>>> > > >> > > >> all
>>> > > >> > > >>>> the
>>> > > >> > > >>>>>>> pending blockers and backports:
>>> > > >> > > >>>>>>>
>>> > > >> > > >>>>>>> *Blockers**:*
>>> > > >> > > >>>>>>>
>>> > > >> > > >>>>>>> Currently all blockers either have an open PR, or have
>>> > been
>>> > > >> > > >> merged.
>>> > > >> > > >>>>> I'll
>>> > > >> > > >>>>>>> highlight below the blockers *that still require
>>> reviewing
>>> > > >> > > >> efforts
>>> > > >> > > >>>> for
>>> > > >> > > >>>>>> them
>>> > > >> > > >>>>>>> to move forward:*
>>> > > >> > > >>>>>>>
>>> > > >> > > >>>>>>> - [FLINK-19909] Flink application in attach mode
>>> could not
>>> > > >> > > >>> terminate
>>> > > >> > > >>>>> when
>>> > > >> > > >>>>>>> the only job is canceled. PR:
>>> > > >> > > >>>>> https://github.com/apache/flink/pull/13911
>>> > > >> > > >>>>>>> - [FLINK-19717] SourceReaderBase.pollNext may return
>>> > > >> END_OF_INPUT
>>> > > >> > > >>> if
>>> > > >> > > >>>>>>> SplitReader.fetch throws. PR:
>>> > > >> > > >>>>> https://github.com/apache/flink/pull/13776
>>> > > >> > > >>>>>>>
>>> > > >> > > >>>>>>> The above PRs currently have no reviews at all yet.
>>> They
>>> > > seem
>>> > > >> to
>>> > > >> > > >>>>> already
>>> > > >> > > >>>>>>> have designated reviewers.
>>> > > >> > > >>>>>>>
>>> > > >> > > >>>>>>>
>>> > > >> > > >>>>>>> *Backports:*
>>> > > >> > > >>>>>>> - FLIP-27 robustness improvement backports: Stephan is
>>> > > >> currently
>>> > > >> > > >>>>> working
>>> > > >> > > >>>>>> on
>>> > > >> > > >>>>>>> backporting several FLIP-27 changes. There are no PRs
>>> yet
>>> > > for
>>> > > >> the
>>> > > >> > > >>>>>>> backported changes.
>>> > > >> > > >>>>>>> - DataStreamUtils.collect() refactorings backport:
>>> Steven
>>> > Wu
>>> > > >> > > >>>> mentioned
>>> > > >> > > >>>>> to
>>> > > >> > > >>>>>>> backport this, but AFAIK this isn't assigned to anyone
>>> > yet.
>>> > > >> > > >>>>>>>
>>> > > >> > > >>>>>>> *ETAs*:
>>> > > >> > > >>>>>>>
>>> > > >> > > >>>>>>> I'd like to request ETAs for the remaining backports,
>>> to
>>> > > >> prevent
>>> > > >> > > >> a
>>> > > >> > > >>>>> creep
>>> > > >> > > >>>>>> in
>>> > > >> > > >>>>>>> the scope of this bugfix release.
>>> > > >> > > >>>>>>>
>>> > > >> > > >>>>>>> We already have Flink users that would benefit from
>>> fixes
>>> > > that
>>> > > >> > > >> have
>>> > > >> > > >>>>> been
>>> > > >> > > >>>>>>> merged to release-1.11 already, so technically
>>> speaking
>>> > the
>>> > > >> > > >>> backports
>>> > > >> > > >>>>>>> should be considered "nice-to-have" (to the best of my
>>> > > >> knowledge
>>> > > >> > > >> of
>>> > > >> > > >>>> the
>>> > > >> > > >>>>>>> changes) and could potentially be moved to a follow-up
>>> > > 1.11.4.
>>> > > >> > > >>>>>>> Most notably, the Stateful Functions project is
>>> already
>>> > > >> waiting
>>> > > >> > > >> on
>>> > > >> > > >>>>> Flink
>>> > > >> > > >>>>>>> 1.11.3 to address critical recovery issues (please
>>> see the
>>> > > >> > > >> StateFun
>>> > > >> > > >>>>> 2.2.1
>>> > > >> > > >>>>>>> release discussion thread [1]).
>>> > > >> > > >>>>>>>
>>> > > >> > > >>>>>>> @Stephan Ewen <se...@apache.org> @Becket Qin <
>>> > > >> > > >> becket.qin@gmail.com
>>> > > >> > > >>>>
>>> > > >> > > >>>>>> could
>>> > > >> > > >>>>>>> you provide an ETA for the FLIP-27 backports? It would
>>> > help
>>> > > to
>>> > > >> > > >> get
>>> > > >> > > >>> a
>>> > > >> > > >>>>>> better
>>> > > >> > > >>>>>>> estimate to decide how we proceed here.
>>> > > >> > > >>>>>>>
>>> > > >> > > >>>>>>> Cheers,
>>> > > >> > > >>>>>>> Gordon
>>> > > >> > > >>>>>>>
>>> > > >> > > >>>>>>> [1]
>>> > > >> > > >>>>>>>
>>> > > >> > > >>>>>>>
>>> > > >> > > >>>>>>
>>> > > >> > > >>>>>
>>> > > >> > > >>>>
>>> > > >> > > >>>
>>> > > >> > > >>
>>> > > >> > >
>>> > > >> >
>>> > > >>
>>> > >
>>> >
>>> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Releasing-StateFun-hotfix-version-2-2-1-td46239.html
>>> > > >> > > >>>>>>>
>>> > > >> > > >>>>>>> On Wed, Nov 4, 2020 at 3:16 PM Tzu-Li (Gordon) Tai <
>>> > > >> > > >>>>> tzulitai@apache.org>
>>> > > >> > > >>>>>>> wrote:
>>> > > >> > > >>>>>>>
>>> > > >> > > >>>>>>>>> The collect() utils can be picked back, I see no
>>> issue
>>> > > with
>>> > > >> > > >>> that
>>> > > >> > > >>>>> (it
>>> > > >> > > >>>>>> is
>>> > > >> > > >>>>>>>> isolated utilities).
>>> > > >> > > >>>>>>>>
>>> > > >> > > >>>>>>>> Just checking on all the requested backports
>>> mentioned in
>>> > > >> this
>>> > > >> > > >>>>> thread,
>>> > > >> > > >>>>>>> and
>>> > > >> > > >>>>>>>> figuring out which ones seem to still be unassigned /
>>> > open.
>>> > > >> > > >>>>>>>>
>>> > > >> > > >>>>>>>> Is someone working on backporting
>>> > > >> > > >>>>>>>>
>>> > > >> > > >>>>>>>
>>> > > >> > > >>>>>>
>>> > > >> > > >>>>>
>>> > > >> > > >>>>
>>> > > >> > > >>>
>>> > > >> > > >>
>>> > > >> > >
>>> > > >> >
>>> > > >>
>>> > >
>>> >
>>> https://github.com/apache/flink/commit/09a7a66b7313fea64817fe960a8da1265b428efc
>>> > > >> > > >>>>>>>> to release-1.11 at the moment?
>>> > > >> > > >>>>>>>>
>>> > > >> > > >>>>>>>> On Wed, Nov 4, 2020 at 1:39 AM Steven Wu <
>>> > > >> stevenz3wu@gmail.com
>>> > > >> > > >>>
>>> > > >> > > >>>>> wrote:
>>> > > >> > > >>>>>>>>
>>> > > >> > > >>>>>>>>> @Stephan Ewen <se...@apache.org> yeah, we can do
>>> that.
>>> > > >> don't
>>> > > >> > > >>>> worry
>>> > > >> > > >>>>>>> about
>>> > > >> > > >>>>>>>>> it. your earlier email had the perfect explanation
>>> on
>>> > why
>>> > > >> file
>>> > > >> > > >>>>> source
>>> > > >> > > >>>>>>>>> shouldn't be backported.
>>> > > >> > > >>>>>>>>>
>>> > > >> > > >>>>>>>>> On Tue, Nov 3, 2020 at 3:37 AM Stephan Ewen <
>>> > > >> sewen@apache.org
>>> > > >> > > >>>
>>> > > >> > > >>>>> wrote:
>>> > > >> > > >>>>>>>>>
>>> > > >> > > >>>>>>>>>> @Steven would it be possible to initially copy
>>> some of
>>> > > the
>>> > > >> > > >>> code
>>> > > >> > > >>>>> into
>>> > > >> > > >>>>>>> the
>>> > > >> > > >>>>>>>>>> iceberg source and later replace it by a
>>> dependency on
>>> > > the
>>> > > >> > > >>> Flink
>>> > > >> > > >>>>>> file
>>> > > >> > > >>>>>>>>>> source?
>>> > > >> > > >>>>>>>>>>
>>> > > >> > > >>>>>>>>>> On Mon, Nov 2, 2020 at 8:33 PM Steven Wu <
>>> > > >> > > >>> stevenz3wu@gmail.com>
>>> > > >> > > >>>>>>> wrote:
>>> > > >> > > >>>>>>>>>>
>>> > > >> > > >>>>>>>>>>> Stephan, thanks a lot for explaining the file
>>> > connector.
>>> > > >> > > >>> that
>>> > > >> > > >>>>>> makes
>>> > > >> > > >>>>>>>>>> sense.
>>> > > >> > > >>>>>>>>>>>
>>> > > >> > > >>>>>>>>>>> I was asking because we were trying to reuse some
>>> of
>>> > the
>>> > > >> > > >>>>>>>>> implementations
>>> > > >> > > >>>>>>>>>> in
>>> > > >> > > >>>>>>>>>>> the file source for Iceberg source. Flink Iceberg
>>> > source
>>> > > >> > > >>> lives
>>> > > >> > > >>>>> in
>>> > > >> > > >>>>>>> the
>>> > > >> > > >>>>>>>>>>> Iceberg repo, which is not possible to code
>>> against
>>> > the
>>> > > >> > > >>> master
>>> > > >> > > >>>>>>> branch
>>> > > >> > > >>>>>>>>> of
>>> > > >> > > >>>>>>>>>>> the Flink code.
>>> > > >> > > >>>>>>>>>>>
>>> > > >> > > >>>>>>>>>>> On Mon, Nov 2, 2020 at 3:31 AM Stephan Ewen <
>>> > > >> > > >>> sewen@apache.org
>>> > > >> > > >>>>>
>>> > > >> > > >>>>>>> wrote:
>>> > > >> > > >>>>>>>>>>>
>>> > > >> > > >>>>>>>>>>>> Hi Steven!
>>> > > >> > > >>>>>>>>>>>>
>>> > > >> > > >>>>>>>>>>>> So far there are no plans to pick back the file
>>> > system
>>> > > >> > > >>>>> connector
>>> > > >> > > >>>>>>>>> code.
>>> > > >> > > >>>>>>>>>>> This
>>> > > >> > > >>>>>>>>>>>> is still evolving and not finalized for 1.12, so
>>> I
>>> > > don't
>>> > > >> > > >>>> feel
>>> > > >> > > >>>>> it
>>> > > >> > > >>>>>>> is
>>> > > >> > > >>>>>>>>> a
>>> > > >> > > >>>>>>>>>>> good
>>> > > >> > > >>>>>>>>>>>> candidate to be backported.
>>> > > >> > > >>>>>>>>>>>> However, with the base connector changes
>>> backported,
>>> > > you
>>> > > >> > > >>>>> should
>>> > > >> > > >>>>>> be
>>> > > >> > > >>>>>>>>> able
>>> > > >> > > >>>>>>>>>>> to
>>> > > >> > > >>>>>>>>>>>> run the file connector code from master against
>>> > 1.11.3.
>>> > > >> > > >>>>>>>>>>>>
>>> > > >> > > >>>>>>>>>>>> The collect() utils can be picked back, I see no
>>> > issue
>>> > > >> > > >>> with
>>> > > >> > > >>>>> that
>>> > > >> > > >>>>>>>>> (it is
>>> > > >> > > >>>>>>>>>>>> isolated utilities).
>>> > > >> > > >>>>>>>>>>>>
>>> > > >> > > >>>>>>>>>>>> Best,
>>> > > >> > > >>>>>>>>>>>> Stephan
>>> > > >> > > >>>>>>>>>>>>
>>> > > >> > > >>>>>>>>>>>>
>>> > > >> > > >>>>>>>>>>>> On Mon, Nov 2, 2020 at 3:02 AM Steven Wu <
>>> > > >> > > >>>>> stevenz3wu@gmail.com>
>>> > > >> > > >>>>>>>>> wrote:
>>> > > >> > > >>>>>>>>>>>>
>>> > > >> > > >>>>>>>>>>>>> Basically, it would be great to get the latest
>>> code
>>> > in
>>> > > >> > > >>> the
>>> > > >> > > >>>>>>>>>>>>> flink-connector-files (FLIP-27).
>>> > > >> > > >>>>>>>>>>>>>
>>> > > >> > > >>>>>>>>>>>>> On Sat, Oct 31, 2020 at 9:57 AM Steven Wu <
>>> > > >> > > >>>>>> stevenz3wu@gmail.com
>>> > > >> > > >>>>>>>>
>>> > > >> > > >>>>>>>>>>> wrote:
>>> > > >> > > >>>>>>>>>>>>>
>>> > > >> > > >>>>>>>>>>>>>> Stephan, it will be great if we can also
>>> backport
>>> > > >> > > >> the
>>> > > >> > > >>>>>>>>>> DataStreamUtils
>>> > > >> > > >>>>>>>>>>>>>> related commits that help with collecting
>>> output
>>> > > >> > > >> from
>>> > > >> > > >>>>>>> unbounded
>>> > > >> > > >>>>>>>>>>>> streams.
>>> > > >> > > >>>>>>>>>>>>>> e.g.
>>> > > >> > > >>>>>>>>>>>>>>
>>> > > >> > > >>>>>>>>>>>>>>
>>> > > >> > > >>>>>>>>>>>>>>
>>> > > >> > > >>>>>>>>>>>>>
>>> > > >> > > >>>>>>>>>>>>
>>> > > >> > > >>>>>>>>>>>
>>> > > >> > > >>>>>>>>>>
>>> > > >> > > >>>>>>>>>
>>> > > >> > > >>>>>>>
>>> > > >> > > >>>>>>
>>> > > >> > > >>>>>
>>> > > >> > > >>>>
>>> > > >> > > >>>
>>> > > >> > > >>
>>> > > >> > >
>>> > > >> >
>>> > > >>
>>> > >
>>> >
>>> https://github.com/apache/flink/commit/09a7a66b7313fea64817fe960a8da1265b428efc
>>> > > >> > > >>>>>>>>>>>>>>
>>> > > >> > > >>>>>>>>>>>>>> I tried to copy and paste the code to unblock
>>> > > >> > > >> myself.
>>> > > >> > > >>>> but
>>> > > >> > > >>>>> it
>>> > > >> > > >>>>>>>>>> quickly
>>> > > >> > > >>>>>>>>>>>> got
>>> > > >> > > >>>>>>>>>>>>>> into the rabbit hole of more and more code.
>>> > > >> > > >>>>>>>>>>>>>>
>>> > > >> > > >>>>>>>>>>>>>> On Fri, Oct 30, 2020 at 11:02 AM Stephan Ewen <
>>> > > >> > > >>>>>>> sewen@apache.org
>>> > > >> > > >>>>>>>>>>
>>> > > >> > > >>>>>>>>>>>> wrote:
>>> > > >> > > >>>>>>>>>>>>>>
>>> > > >> > > >>>>>>>>>>>>>>> I have started with backporting the source API
>>> > > >> > > >>> changes.
>>> > > >> > > >>>>>> Some
>>> > > >> > > >>>>>>>>> minor
>>> > > >> > > >>>>>>>>>>>>>>> conflicts to solve, will need a bit more to
>>> finish
>>> > > >> > > >>>> this.
>>> > > >> > > >>>>>>>>>>>>>>>
>>> > > >> > > >>>>>>>>>>>>>>> On Fri, Oct 30, 2020 at 7:25 AM Tzu-Li
>>> (Gordon)
>>> > > >> > > >> Tai <
>>> > > >> > > >>>>>>>>>>>>> tzulitai@apache.org>
>>> > > >> > > >>>>>>>>>>>>>>> wrote:
>>> > > >> > > >>>>>>>>>>>>>>>
>>> > > >> > > >>>>>>>>>>>>>>>> @Stephan Ewen <se...@apache.org>
>>> > > >> > > >>>>>>>>>>>>>>>> Are there already plans or ongoing efforts
>>> for
>>> > > >> > > >>>>>> backporting
>>> > > >> > > >>>>>>>>> the
>>> > > >> > > >>>>>>>>>>> list
>>> > > >> > > >>>>>>>>>>>> of
>>> > > >> > > >>>>>>>>>>>>>>>> FLIP-27 changes that you posted?
>>> > > >> > > >>>>>>>>>>>>>>>>
>>> > > >> > > >>>>>>>>>>>>>>>> On Thu, Oct 29, 2020 at 7:08 PM Xintong Song
>>> <
>>> > > >> > > >>>>>>>>>>> tonysong820@gmail.com
>>> > > >> > > >>>>>>>>>>>>>
>>> > > >> > > >>>>>>>>>>>>>>>> wrote:
>>> > > >> > > >>>>>>>>>>>>>>>>
>>> > > >> > > >>>>>>>>>>>>>>>>> Hi folks,
>>> > > >> > > >>>>>>>>>>>>>>>>>
>>> > > >> > > >>>>>>>>>>>>>>>>> Just to provide some updates concerning the
>>> > > >> > > >> status
>>> > > >> > > >>>> on
>>> > > >> > > >>>>>> the
>>> > > >> > > >>>>>>>>>>>>>>>>> test instabilities.
>>> > > >> > > >>>>>>>>>>>>>>>>>
>>> > > >> > > >>>>>>>>>>>>>>>>> Currently, we have 30 unresolved tickets
>>> labeled
>>> > > >> > > >>>> with
>>> > > >> > > >>>>>>>>> `Affects
>>> > > >> > > >>>>>>>>>>>>> Version`
>>> > > >> > > >>>>>>>>>>>>>>>>> 1.11.x.
>>> > > >> > > >>>>>>>>>>>>>>>>>
>>> > > >> > > >>>>>>>>>>>>>>>>>
>>> > > >> > > >>>>>>>>>>>>>>>
>>> > > >> > > >>>>>>>>>>>>>
>>> > > >> > > >>>>>>>>>>>>
>>> > > >> > > >>>>>>>>>>>
>>> > > >> > > >>>>>>>>>>
>>> > > >> > > >>>>>>>>>
>>> > > >> > > >>>>>>>
>>> > > >> > > >>>>>>
>>> > > >> > > >>>>>
>>> > > >> > > >>>>
>>> > > >> > > >>>
>>> > > >> > > >>
>>> > > >> > >
>>> > > >> >
>>> > > >>
>>> > >
>>> >
>>> https://issues.apache.org/jira/browse/FLINK-19775?filter=12348580&jql=project%20%3D%20FLINK%20AND%20resolution%20%3D%20Unresolved%20AND%20affectedVersion%20in%20(1.11.0%2C%201.11.1%2C%201.11.2%2C%201.11.3)%20AND%20labels%20%3D%20test-stability%20ORDER%20BY%20created%20DESC
>>> > > >> > > >>>>>>>>>>>>>>>>>
>>> > > >> > > >>>>>>>>>>>>>>>>> Among the 30 tickets, 11 of them are:
>>> > > >> > > >>>>>>>>>>>>>>>>> - Have occured in the recent 3 months
>>> > > >> > > >>>>>>>>>>>>>>>>> - Not confirmed to be pure testability
>>> issues
>>> > > >> > > >>>>>>>>>>>>>>>>> - Not confirmed to be rare condition cases
>>> > > >> > > >>>>>>>>>>>>>>>>>
>>> > > >> > > >>>>>>>>>>>>>>>>> It would be nice if someone familiar with
>>> these
>>> > > >> > > >>>>>> components
>>> > > >> > > >>>>>>>>> can
>>> > > >> > > >>>>>>>>>>>> take a
>>> > > >> > > >>>>>>>>>>>>>>> look
>>> > > >> > > >>>>>>>>>>>>>>>>> into these issues.
>>> > > >> > > >>>>>>>>>>>>>>>>>
>>> > > >> > > >>>>>>>>>>>>>>>>> -
>>> > > >> > > >>> https://issues.apache.org/jira/browse/FLINK-17159
>>> > > >> > > >>>>>> (ES6)
>>> > > >> > > >>>>>>>>>>>>>>>>> -
>>> > > >> > > >>> https://issues.apache.org/jira/browse/FLINK-17912
>>> > > >> > > >>>>>>> (Kafka)
>>> > > >> > > >>>>>>>>>>>>>>>>> -
>>> > > >> > > >>> https://issues.apache.org/jira/browse/FLINK-17949
>>> > > >> > > >>>>>>> (Kafka)
>>> > > >> > > >>>>>>>>>>>>>>>>> ⁃
>>> > > >> > > >>> https://issues.apache.org/jira/browse/FLINK-18444
>>> > > >> > > >>>>>>> (Kafka)
>>> > > >> > > >>>>>>>>>>>>>>>>> -
>>> > > >> > > >>> https://issues.apache.org/jira/browse/FLINK-18634
>>> > > >> > > >>>>>>> (Kafka)
>>> > > >> > > >>>>>>>>>>>>>>>>> -
>>> > > >> > > >>> https://issues.apache.org/jira/browse/FLINK-18648
>>> > > >> > > >>>>>>> (Kafka)
>>> > > >> > > >>>>>>>>>>>>>>>>> -
>>> > > >> > > >>> https://issues.apache.org/jira/browse/FLINK-18807
>>> > > >> > > >>>>>>> (Kafka)
>>> > > >> > > >>>>>>>>>>>>>>>>> -
>>> > > >> > > >>> https://issues.apache.org/jira/browse/FLINK-19369
>>> > > >> > > >>>>>>>>>>>> (BlobClientTest)
>>> > > >> > > >>>>>>>>>>>>>>>>> -
>>> > > >> > > >>> https://issues.apache.org/jira/browse/FLINK-19436
>>> > > >> > > >>>>>>> (TPCDS)
>>> > > >> > > >>>>>>>>>>>>>>>>> -
>>> > > >> > > >>> https://issues.apache.org/jira/browse/FLINK-19690
>>> > > >> > > >>>>>>>>>>>> (Format/Parquet)
>>> > > >> > > >>>>>>>>>>>>>>>>> -
>>> > > >> > > >>> https://issues.apache.org/jira/browse/FLINK-19775
>>> > > >> > > >>>>>>>>>>>>>>>>> (SystemProcessingTimeServiceTest)
>>> > > >> > > >>>>>>>>>>>>>>>>>
>>> > > >> > > >>>>>>>>>>>>>>>>> Thank you~
>>> > > >> > > >>>>>>>>>>>>>>>>>
>>> > > >> > > >>>>>>>>>>>>>>>>> Xintong Song
>>> > > >> > > >>>>>>>>>>>>>>>>>
>>> > > >> > > >>>>>>>>>>>>>>>>>
>>> > > >> > > >>>>>>>>>>>>>>>>>
>>> > > >> > > >>>>>>>>>>>>>>>>> On Thu, Oct 29, 2020 at 10:21 AM Jingsong
>>> Li <
>>> > > >> > > >>>>>>>>>>>> jingsonglee0@gmail.com
>>> > > >> > > >>>>>>>>>>>>>>
>>> > > >> > > >>>>>>>>>>>>>>>>> wrote:
>>> > > >> > > >>>>>>>>>>>>>>>>>
>>> > > >> > > >>>>>>>>>>>>>>>>>> +1 to backport the FLIP-27 adjustments to
>>> > > >> > > >>> 1.11.x.
>>> > > >> > > >>>>>>>>>>>>>>>>>>
>>> > > >> > > >>>>>>>>>>>>>>>>>> If possible, that would be great. Many
>>> people
>>> > > >> > > >>> are
>>> > > >> > > >>>>>>> looking
>>> > > >> > > >>>>>>>>>>> forward
>>> > > >> > > >>>>>>>>>>>>> to
>>> > > >> > > >>>>>>>>>>>>>>> the
>>> > > >> > > >>>>>>>>>>>>>>>>>> FLIP-27 interface, but they don't want to
>>> take
>>> > > >> > > >>> the
>>> > > >> > > >>>>>> risk
>>> > > >> > > >>>>>>> to
>>> > > >> > > >>>>>>>>>>>> upgrade
>>> > > >> > > >>>>>>>>>>>>> to
>>> > > >> > > >>>>>>>>>>>>>>>>> 1.12
>>> > > >> > > >>>>>>>>>>>>>>>>>> (And wait 1.12). After all, 1.11 is a
>>> > > >> > > >> relatively
>>> > > >> > > >>>>>> stable
>>> > > >> > > >>>>>>>>>>> version.
>>> > > >> > > >>>>>>>>>>>>>>>>>>
>>> > > >> > > >>>>>>>>>>>>>>>>>> Best,
>>> > > >> > > >>>>>>>>>>>>>>>>>> Jingsong
>>> > > >> > > >>>>>>>>>>>>>>>>>>
>>> > > >> > > >>>>>>>>>>>>>>>>>> On Thu, Oct 29, 2020 at 1:24 AM Stephan
>>> Ewen <
>>> > > >> > > >>>>>>>>>> sewen@apache.org
>>> > > >> > > >>>>>>>>>>>>
>>> > > >> > > >>>>>>>>>>>>>>> wrote:
>>> > > >> > > >>>>>>>>>>>>>>>>>>
>>> > > >> > > >>>>>>>>>>>>>>>>>>> Thanks for starting this.
>>> > > >> > > >>>>>>>>>>>>>>>>>>>
>>> > > >> > > >>>>>>>>>>>>>>>>>>> +1 form my side to backport the FLIP-27
>>> > > >> > > >>>>> adjustments
>>> > > >> > > >>>>>> to
>>> > > >> > > >>>>>>>>>>> 1.11.x.
>>> > > >> > > >>>>>>>>>>>>>>>>>>>
>>> > > >> > > >>>>>>>>>>>>>>>>>>> There were quite a few changes, and I
>>> think
>>> > > >> > > >> we
>>> > > >> > > >>>>> need
>>> > > >> > > >>>>>> to
>>> > > >> > > >>>>>>>>>>>>> cherry-pick
>>> > > >> > > >>>>>>>>>>>>>>>>> them
>>> > > >> > > >>>>>>>>>>>>>>>>>> all
>>> > > >> > > >>>>>>>>>>>>>>>>>>> to not get some inconsistent mix of
>>> changes
>>> > > >> > > >>> and
>>> > > >> > > >>>>> many
>>> > > >> > > >>>>>>>>> merge
>>> > > >> > > >>>>>>>>>>>>>>> conflicts.
>>> > > >> > > >>>>>>>>>>>>>>>>>>> I made a list below of what we need to add
>>> > > >> > > >> to
>>> > > >> > > >>>>>>>>>> "release-1.11".
>>> > > >> > > >>>>>>>>>>>>>>>>>>>
>>> > > >> > > >>>>>>>>>>>>>>>>>>> *#### Core Source API Changes to backport
>>> > > >> > > >> (in
>>> > > >> > > >>>>>> REVERSE
>>> > > >> > > >>>>>>>>>> order)*
>>> > > >> > > >>>>>>>>>>>>>>>>>>>
>>> > > >> > > >>>>>>>>>>>>>>>>>>>  (Use: "git log
>>> > > >> > > >>>>>>>>>>>>>>>>>>>
>>> > > >> > > >>>>>>>>>>>>
>>> > > >> > > >>>>>
>>> > > flink-core/src/main/java/org/apache/flink/api/connector/source")
>>> > > >> > > >>>>>>>>>>>>>>>>>>>
>>> > > >> > > >>>>>>>>>>>>>>>>>>> commit
>>> > > >> > > >>> 162c072e9265a7b6dd9d6f5459eb7974091c4c4e
>>> > > >> > > >>>>>>>>>>>>>>>>>>> [FLINK-19492][core] Consolidate Source
>>> > > >> > > >> Events
>>> > > >> > > >>>>>> between
>>> > > >> > > >>>>>>>>>> Source
>>> > > >> > > >>>>>>>>>>>> API
>>> > > >> > > >>>>>>>>>>>>>>> and
>>> > > >> > > >>>>>>>>>>>>>>>>>> Split
>>> > > >> > > >>>>>>>>>>>>>>>>>>> Reader API
>>> > > >> > > >>>>>>>>>>>>>>>>>>>
>>> > > >> > > >>>>>>>>>>>>>>>>>>> commit
>>> > > >> > > >>> 428fbbad85379f71a9fbf1a1ab3e3dcfa9006436
>>> > > >> > > >>>>>>>>>>>>>>>>>>> [FLINK-19205][core] Add access to
>>> > > >> > > >>> configuration
>>> > > >> > > >>>>> and
>>> > > >> > > >>>>>>>>>> hostname
>>> > > >> > > >>>>>>>>>>> in
>>> > > >> > > >>>>>>>>>>>>> the
>>> > > >> > > >>>>>>>>>>>>>>>>>>> SourceReaderContext
>>> > > >> > > >>>>>>>>>>>>>>>>>>>
>>> > > >> > > >>>>>>>>>>>>>>>>>>> commit
>>> > > >> > > >>> 723e1790744ddc78d19c7c978442af1383f38d33
>>> > > >> > > >>>>>>>>>>>>>>>>>>> [hotfix][core] Add to Source Enumerator
>>> > > >> > > >>>>> convenience
>>> > > >> > > >>>>>>>>> methods
>>> > > >> > > >>>>>>>>>>> to
>>> > > >> > > >>>>>>>>>>>>>>> assign
>>> > > >> > > >>>>>>>>>>>>>>>>>>> single split
>>> > > >> > > >>>>>>>>>>>>>>>>>>>
>>> > > >> > > >>>>>>>>>>>>>>>>>>> *#### Connector Base Changes to Backport
>>> (in
>>> > > >> > > >>>>> REVERSE
>>> > > >> > > >>>>>>>>>> order)*
>>> > > >> > > >>>>>>>>>>>>>>>>>>>
>>> > > >> > > >>>>>>>>>>>>>>>>>>>  (Use: "git log
>>> > > >> > > >>>>>>> flink-connectors/flink-connector-base")
>>> > > >> > > >>>>>>>>>>>>>>>>>>>
>>> > > >> > > >>>>>>>>>>>>>>>>>>> commit
>>> > > >> > > >>> 401f56fe9d6b0271260edf9787cdcbfe4d03874d
>>> > > >> > > >>>>>>>>>>>>>>>>>>> [FLINK-19427][FLINK-19489][tests] Fix test
>>> > > >> > > >>>>>> conditions
>>> > > >> > > >>>>>>>>> for
>>> > > >> > > >>>>>>>>>>>>>>>>>>>
>>> > > >> > > >>>>>>>
>>> 'SplitFetcherTest.testNotifiesWhenGoingIdleConcurrent()'
>>> > > >> > > >>>>>>>>>>>>>>>>>>>
>>> > > >> > > >>>>>>>>>>>>>>>>>>> commit
>>> > > >> > > >>> 68c5c2ff779d82a1ff81ffaf60d8a1b283797db1
>>> > > >> > > >>>>>>>>>>>>>>>>>>> [FLINK-19448][connector base] Explicitly
>>> > > >> > > >> check
>>> > > >> > > >>>> for
>>> > > >> > > >>>>>>>>>>> un-expected
>>> > > >> > > >>>>>>>>>>>>>>>>> condition
>>> > > >> > > >>>>>>>>>>>>>>>>>>> that would leave an inconsistent state
>>> > > >> > > >>>>>>>>>>>>>>>>>>>
>>> > > >> > > >>>>>>>>>>>>>>>>>>> commit
>>> > > >> > > >>> 162c072e9265a7b6dd9d6f5459eb7974091c4c4e
>>> > > >> > > >>>>>>>>>>>>>>>>>>> [FLINK-19492][core] Consolidate Source
>>> > > >> > > >> Events
>>> > > >> > > >>>>>> between
>>> > > >> > > >>>>>>>>>> Source
>>> > > >> > > >>>>>>>>>>>> API
>>> > > >> > > >>>>>>>>>>>>>>> and
>>> > > >> > > >>>>>>>>>>>>>>>>>> Split
>>> > > >> > > >>>>>>>>>>>>>>>>>>> Reader API
>>> > > >> > > >>>>>>>>>>>>>>>>>>>
>>> > > >> > > >>>>>>>>>>>>>>>>>>> commit
>>> > > >> > > >>> c1ca7a4c7c21ec8868c14cdddd43c559625b794c
>>> > > >> > > >>>>>>>>>>>>>>>>>>> [refactor][tests] Move some source test
>>> > > >> > > >> utils
>>> > > >> > > >>>> from
>>> > > >> > > >>>>>>>>>>>>>>>>> flink-connector-base
>>> > > >> > > >>>>>>>>>>>>>>>>>> to
>>> > > >> > > >>>>>>>>>>>>>>>>>>> flink-core
>>> > > >> > > >>>>>>>>>>>>>>>>>>>
>>> > > >> > > >>>>>>>>>>>>>>>>>>> commit
>>> > > >> > > >>> ee5c4c211c35c70d28252363bbc8400453609977
>>> > > >> > > >>>>>>>>>>>>>>>>>>> [FLINK-19251][connectors] Avoid confusing
>>> > > >> > > >>> queue
>>> > > >> > > >>>>>>>>> handling in
>>> > > >> > > >>>>>>>>>>>>>>>>>>> "SplitReader.handleSplitsChanges()"
>>> > > >> > > >>>>>>>>>>>>>>>>>>>
>>> > > >> > > >>>>>>>>>>>>>>>>>>> commit
>>> > > >> > > >>> 5abef56b2bf85bcac786f6b16b6899b6cced7176
>>> > > >> > > >>>>>>>>>>>>>>>>>>> [FLINK-19250][connectors] Fix error
>>> > > >> > > >>> propagation
>>> > > >> > > >>>> in
>>> > > >> > > >>>>>>>>>> connector
>>> > > >> > > >>>>>>>>>>>> base
>>> > > >> > > >>>>>>>>>>>>>>>>>>> (SplitFetcherManager).
>>> > > >> > > >>>>>>>>>>>>>>>>>>>
>>> > > >> > > >>>>>>>>>>>>>>>>>>> commit
>>> > > >> > > >>> 8fcca837c55a9216595ee4c03038b52747098dbb
>>> > > >> > > >>>>>>>>>>>>>>>>>>> [hotfix][connectors] Improve JavaDocs for
>>> > > >> > > >>>>>>>>>>>>>>> SingleThreadFetcherManager
>>> > > >> > > >>>>>>>>>>>>>>>>>>>
>>> > > >> > > >>>>>>>>>>>>>>>>>>> commit
>>> > > >> > > >>> 4700bb5dde3303cbe98882f6beb7379425717b01
>>> > > >> > > >>>>>>>>>>>>>>>>>>> [FLINK-19225][connectors] Various small
>>> > > >> > > >>>>> improvements
>>> > > >> > > >>>>>>> to
>>> > > >> > > >>>>>>>>>>>>>>>>> SourceReaderBase
>>> > > >> > > >>>>>>>>>>>>>>>>>>> (part 2)
>>> > > >> > > >>>>>>>>>>>>>>>>>>>
>>> > > >> > > >>>>>>>>>>>>>>>>>>> commit
>>> > > >> > > >>> 12261c6b7ed6478a9b9f6a69cb58246b83cab9b7
>>> > > >> > > >>>>>>>>>>>>>>>>>>> [FLINK-17393][connectors] (follow-up)
>>> Wakeup
>>> > > >> > > >>> the
>>> > > >> > > >>>>>>>>>>> SplitFetchers
>>> > > >> > > >>>>>>>>>>>>> more
>>> > > >> > > >>>>>>>>>>>>>>>>>>> elegantly.
>>> > > >> > > >>>>>>>>>>>>>>>>>>>
>>> > > >> > > >>>>>>>>>>>>>>>>>>> commit
>>> > > >> > > >>> c60aaff0249bfd6b5871b7f82e03efc487a54d6b
>>> > > >> > > >>>>>>>>>>>>>>>>>>> [hotfix][tests] Extend test coverage for
>>> > > >> > > >>>>>>>>>>>>>>> FutureCompletingBlockingQueue
>>> > > >> > > >>>>>>>>>>>>>>>>>>>
>>> > > >> > > >>>>>>>>>>>>>>>>>>> commit
>>> > > >> > > >>> cef8a587d7fd2fe64cc644da5ed095d82e46f631
>>> > > >> > > >>>>>>>>>>>>>>>>>>> [FLINK-19245][connectors] Set default
>>> > > >> > > >> capacity
>>> > > >> > > >>>> for
>>> > > >> > > >>>>>>>>>>>>>>>>>>> FutureCompletingBlockingQueue.
>>> > > >> > > >>>>>>>>>>>>>>>>>>>
>>> > > >> > > >>>>>>>>>>>>>>>>>>> commit
>>> > > >> > > >>> 4ea95782b4c6a2538153d4d16ad3f4839c7de0fb
>>> > > >> > > >>>>>>>>>>>>>>>>>>> [FLINK-19223][connectors] Simplify
>>> > > >> > > >>> Availability
>>> > > >> > > >>>>>> Future
>>> > > >> > > >>>>>>>>>> Model
>>> > > >> > > >>>>>>>>>>> in
>>> > > >> > > >>>>>>>>>>>>>>> Base
>>> > > >> > > >>>>>>>>>>>>>>>>>>> Connector
>>> > > >> > > >>>>>>>>>>>>>>>>>>>
>>> > > >> > > >>>>>>>>>>>>>>>>>>> commit
>>> > > >> > > >>> 511857049ba30c8ff0ee56da551fa4a479dc583e
>>> > > >> > > >>>>>>>>>>>>>>>>>>> [FLINK-18128][connectors] Ensure idle
>>> split
>>> > > >> > > >>>>> fetchers
>>> > > >> > > >>>>>>>>> lead
>>> > > >> > > >>>>>>>>>> to
>>> > > >> > > >>>>>>>>>>>>>>>>> availability
>>> > > >> > > >>>>>>>>>>>>>>>>>>> notifications.
>>> > > >> > > >>>>>>>>>>>>>>>>>>>
>>> > > >> > > >>>>>>>>>>>>>>>>>>> commit
>>> > > >> > > >>> a8206467af0830dcb89623ea068b5ca3b3450c92
>>> > > >> > > >>>>>>>>>>>>>>>>>>> [refactor][core] Eagerly initialize the
>>> > > >> > > >>>> FetchTask
>>> > > >> > > >>>>> to
>>> > > >> > > >>>>>>>>>> support
>>> > > >> > > >>>>>>>>>>>>> proper
>>> > > >> > > >>>>>>>>>>>>>>>>> unit
>>> > > >> > > >>>>>>>>>>>>>>>>>>> testing
>>> > > >> > > >>>>>>>>>>>>>>>>>>>
>>> > > >> > > >>>>>>>>>>>>>>>>>>> commit
>>> > > >> > > >>> 3b2f54bcb437f98e6137c904045cc51072b5c06b
>>> > > >> > > >>>>>>>>>>>>>>>>>>> [hotfix][tests] Move constants in
>>> > > >> > > >>>> SplitFetcherTest
>>> > > >> > > >>>>>>>>> relevant
>>> > > >> > > >>>>>>>>>>> to
>>> > > >> > > >>>>>>>>>>>>> only
>>> > > >> > > >>>>>>>>>>>>>>>>> one
>>> > > >> > > >>>>>>>>>>>>>>>>>>> test into test method
>>> > > >> > > >>>>>>>>>>>>>>>>>>>
>>> > > >> > > >>>>>>>>>>>>>>>>>>> commit
>>> > > >> > > >>> d7625760a75a508bf05bcddc380bb4d62ee1743e
>>> > > >> > > >>>>>>>>>>>>>>>>>>> [FLINK-19225][connectors] Various small
>>> > > >> > > >>>>> improvements
>>> > > >> > > >>>>>>> to
>>> > > >> > > >>>>>>>>>>>>>>>>> SourceReaderBase
>>> > > >> > > >>>>>>>>>>>>>>>>>>>
>>> > > >> > > >>>>>>>>>>>>>>>>>>> commit
>>> > > >> > > >>> a5b0d3297748c1be47ad579a88f24df2255a8df1
>>> > > >> > > >>>>>>>>>>>>>>>>>>> [FLINK-17393][connectors] Wakeup the
>>> > > >> > > >>>> SplitFetchers
>>> > > >> > > >>>>>>> more
>>> > > >> > > >>>>>>>>>>>>> elegantly.
>>> > > >> > > >>>>>>>>>>>>>>>>>>>
>>> > > >> > > >>>>>>>>>>>>>>>>>>> commit
>>> > > >> > > >>> f42a3ebc3e81a034b7221a803c153636fef34903
>>> > > >> > > >>>>>>>>>>>>>>>>>>> [FLINK-18680][connectors] Make connector
>>> > > >> > > >> base
>>> > > >> > > >>>>>>>>>>>> RecordsWithSplitIds
>>> > > >> > > >>>>>>>>>>>>>>> more
>>> > > >> > > >>>>>>>>>>>>>>>>>>> lightweight.
>>> > > >> > > >>>>>>>>>>>>>>>>>>>
>>> > > >> > > >>>>>>>>>>>>>>>>>>> commit
>>> > > >> > > >>> e3d273de822b085183d09b275a445879ff94b350
>>> > > >> > > >>>>>>>>>>>>>>>>>>> [FLINK-19162][connectors] Add 'recycle()'
>>> to
>>> > > >> > > >>> the
>>> > > >> > > >>>>>>>>>>>>>>> RecordsWithSplitIds
>>> > > >> > > >>>>>>>>>>>>>>>>> to
>>> > > >> > > >>>>>>>>>>>>>>>>>>> support reuse of heavy objects.
>>> > > >> > > >>>>>>>>>>>>>>>>>>>
>>> > > >> > > >>>>>>>>>>>>>>>>>>> commit
>>> > > >> > > >>> 8ebc464c2520453a70001cd712abc8dee6ee89e0
>>> > > >> > > >>>>>>>>>>>>>>>>>>> [hotfix][testing] Add a set of
>>> > > >> > > >> parameterizable
>>> > > >> > > >>>>>> testing
>>> > > >> > > >>>>>>>>>> mocks
>>> > > >> > > >>>>>>>>>>>> for
>>> > > >> > > >>>>>>>>>>>>>>> the
>>> > > >> > > >>>>>>>>>>>>>>>>>> Split
>>> > > >> > > >>>>>>>>>>>>>>>>>>> Reader API
>>> > > >> > > >>>>>>>>>>>>>>>>>>>
>>> > > >> > > >>>>>>>>>>>>>>>>>>> commit
>>> > > >> > > >>> 930a07438be1185388d7150640f294dfe2a5d378
>>> > > >> > > >>>>>>>>>>>>>>>>>>> [hotfix][connectors] Add RequestSplitEvent
>>> > > >> > > >> to
>>> > > >> > > >>>>>>>>>>>>>>> 'flink-connector-base'
>>> > > >> > > >>>>>>>>>>>>>>>>>>>
>>> > > >> > > >>>>>>>>>>>>>>>>>>> commit
>>> > > >> > > >>> e945ce8a933bc378844782f784ca473c767ca159
>>> > > >> > > >>>>>>>>>>>>>>>>>>> [refactor][connectors] Backport of the
>>> > > >> > > >>>>>> connector-base
>>> > > >> > > >>>>>>>>>>> exception
>>> > > >> > > >>>>>>>>>>>>>>>>> handling
>>> > > >> > > >>>>>>>>>>>>>>>>>>> from the Kafka Connector Pull Request
>>> > > >> > > >>>>>>>>>>>>>>>>>>>
>>> > > >> > > >>>>>>>>>>>>>>>>>>>
>>> > > >> > > >>>>>>>>>>>>>>>>>>> On Wed, Oct 28, 2020 at 5:49 PM Steven Wu
>>> <
>>> > > >> > > >>>>>>>>>>>> stevenz3wu@gmail.com>
>>> > > >> > > >>>>>>>>>>>>>>>>> wrote:
>>> > > >> > > >>>>>>>>>>>>>>>>>>>
>>> > > >> > > >>>>>>>>>>>>>>>>>>>> I would love to see this FLIP-27 source
>>> > > >> > > >>>>> interface
>>> > > >> > > >>>>>>>>>>> improvement
>>> > > >> > > >>>>>>>>>>>>> [1]
>>> > > >> > > >>>>>>>>>>>>>>>>> made
>>> > > >> > > >>>>>>>>>>>>>>>>>> to
>>> > > >> > > >>>>>>>>>>>>>>>>>>>> 1.11.3.
>>> > > >> > > >>>>>>>>>>>>>>>>>>>>
>>> > > >> > > >>>>>>>>>>>>>>>>>>>> [1]
>>> > > >> > > >>>>>>> https://issues.apache.org/jira/browse/FLINK-19698
>>> > > >> > > >>>>>>>>>>>>>>>>>>>>
>>> > > >> > > >>>>>>>>>>>>>>>>>>>> On Wed, Oct 28, 2020 at 12:32 AM Tzu-Li
>>> > > >> > > >>>> (Gordon)
>>> > > >> > > >>>>>>> Tai <
>>> > > >> > > >>>>>>>>>>>>>>>>>>> tzulitai@apache.org>
>>> > > >> > > >>>>>>>>>>>>>>>>>>>> wrote:
>>> > > >> > > >>>>>>>>>>>>>>>>>>>>
>>> > > >> > > >>>>>>>>>>>>>>>>>>>>> Thanks for the replies so far!
>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>
>>> > > >> > > >>>>>>>>>>>>>>>>>>>>> Just to provide a brief update on the
>>> > > >> > > >>> status
>>> > > >> > > >>>>> of
>>> > > >> > > >>>>>>>>>> blockers
>>> > > >> > > >>>>>>>>>>>> for
>>> > > >> > > >>>>>>>>>>>>>>>>> 1.11.3
>>> > > >> > > >>>>>>>>>>>>>>>>>> so
>>> > > >> > > >>>>>>>>>>>>>>>>>>>> far:
>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>
>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>
>>> > > >> > > >>>>>>>>>>>>>>>>>>>>> *PR opened, pending reviewer*-
>>> > > >> > > >>> [FLINK-19717]
>>> > > >> > > >>>>>>>>>>>>>>>>>> SourceReaderBase.pollNext
>>> > > >> > > >>>>>>>>>>>>>>>>>>>> may
>>> > > >> > > >>>>>>>>>>>>>>>>>>>>> return END_OF_INPUT if SplitReader.fetch
>>> > > >> > > >>>>> throws
>>> > > >> > > >>>>>> (
>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>
>>> > > >> > > >>> https://github.com/apache/flink/pull/13776)
>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>
>>> > > >> > > >>>>>>>>>>>>>>>>>>>>> *PR opened, reviewed + close to being
>>> > > >> > > >>>> merged*
>>> > > >> > > >>>>>>>>>>>>>>>>>>>>> - [FLINK-19741] Timer service should
>>> > > >> > > >> skip
>>> > > >> > > >>>>>>> restoring
>>> > > >> > > >>>>>>>>>> from
>>> > > >> > > >>>>>>>>>>>> raw
>>> > > >> > > >>>>>>>>>>>>>>> keyed
>>> > > >> > > >>>>>>>>>>>>>>>>>>> stream
>>> > > >> > > >>>>>>>>>>>>>>>>>>>>> if it isn't the writer (
>>> > > >> > > >>>>>>>>>>>>>>>>> https://github.com/apache/flink/pull/13761)
>>> > > >> > > >>>>>>>>>>>>>>>>>>>>> - [FLINK-19748] Raw keyed stream key
>>> > > >> > > >> group
>>> > > >> > > >>>>>>> iterator
>>> > > >> > > >>>>>>>>>>> should
>>> > > >> > > >>>>>>>>>>>> be
>>> > > >> > > >>>>>>>>>>>>>>>>>> skipping
>>> > > >> > > >>>>>>>>>>>>>>>>>>>>> unwritten key groups (
>>> > > >> > > >>>>>>>>>>>>>>> https://github.com/apache/flink/pull/13772)
>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>
>>> > > >> > > >>>>>>>>>>>>>>>>>>>>> *Merged*
>>> > > >> > > >>>>>>>>>>>>>>>>>>>>> - [FLINK-19154] Application mode deletes
>>> > > >> > > >>> HA
>>> > > >> > > >>>>> data
>>> > > >> > > >>>>>>> in
>>> > > >> > > >>>>>>>>>> case
>>> > > >> > > >>>>>>>>>>> of
>>> > > >> > > >>>>>>>>>>>>>>>>> suspended
>>> > > >> > > >>>>>>>>>>>>>>>>>>>>> ZooKeeper connection
>>> > > >> > > >>>>>>>>>>>>>>>>>>>>> - [FLINK-19569] Upgrade ICU4J to 67.1+
>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>
>>> > > >> > > >>>>>>>>>>>>>>>>>>>>> Right now as it seems, progress is
>>> > > >> > > >> mainly
>>> > > >> > > >>>>>> blocked
>>> > > >> > > >>>>>>>>> on a
>>> > > >> > > >>>>>>>>>>>>> reviewer
>>> > > >> > > >>>>>>>>>>>>>>>>> for
>>> > > >> > > >>>>>>>>>>>>>>>>>>>>> FLINK-19717.
>>> > > >> > > >>>>>>>>>>>>>>>>>>>>> Meanwhile, Xintong is keeping an eye on
>>> > > >> > > >>> test
>>> > > >> > > >>>>>>>>>>> instabilities
>>> > > >> > > >>>>>>>>>>>>> [1]
>>> > > >> > > >>>>>>>>>>>>>>> to
>>> > > >> > > >>>>>>>>>>>>>>>>> see
>>> > > >> > > >>>>>>>>>>>>>>>>>>> if
>>> > > >> > > >>>>>>>>>>>>>>>>>>>>> there are any fixes that should be
>>> > > >> > > >> applied
>>> > > >> > > >>>> to
>>> > > >> > > >>>>>>>>>>>> `release-1.11`.
>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>
>>> > > >> > > >>>>>>>>>>>>>>>>>>>>> This is also a reminder, that if there
>>> > > >> > > >> are
>>> > > >> > > >>>>> other
>>> > > >> > > >>>>>>>>>> blockers
>>> > > >> > > >>>>>>>>>>>>> that
>>> > > >> > > >>>>>>>>>>>>>>> we
>>> > > >> > > >>>>>>>>>>>>>>>>>> need
>>> > > >> > > >>>>>>>>>>>>>>>>>>> to
>>> > > >> > > >>>>>>>>>>>>>>>>>>>>> be aware of, or a need to re-establish
>>> > > >> > > >>>>> estimated
>>> > > >> > > >>>>>>>>> time
>>> > > >> > > >>>>>>>>>> for
>>> > > >> > > >>>>>>>>>>>>>>> getting
>>> > > >> > > >>>>>>>>>>>>>>>>>> fixes
>>> > > >> > > >>>>>>>>>>>>>>>>>>>> in
>>> > > >> > > >>>>>>>>>>>>>>>>>>>>> and delay the RC for 1.11.3, please do
>>> > > >> > > >> let
>>> > > >> > > >>>> us
>>> > > >> > > >>>>>>> know!
>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>
>>> > > >> > > >>>>>>>>>>>>>>>>>>>>> Cheers,
>>> > > >> > > >>>>>>>>>>>>>>>>>>>>> Gordon
>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>
>>> > > >> > > >>>>>>>>>>>>>>>>>>>>> [1]
>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>
>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>
>>> > > >> > > >>>>>>>>>>>>>>>>>>>>
>>> > > >> > > >>>>>>>>>>>>>>>>>>>
>>> > > >> > > >>>>>>>>>>>>>>>>>>
>>> > > >> > > >>>>>>>>>>>>>>>>>
>>> > > >> > > >>>>>>>>>>>>>>>
>>> > > >> > > >>>>>>>>>>>>>
>>> > > >> > > >>>>>>>>>>>>
>>> > > >> > > >>>>>>>>>>>
>>> > > >> > > >>>>>>>>>>
>>> > > >> > > >>>>>>>>>
>>> > > >> > > >>>>>>>
>>> > > >> > > >>>>>>
>>> > > >> > > >>>>>
>>> > > >> > > >>>>
>>> > > >> > > >>>
>>> > > >> > > >>
>>> > > >> > >
>>> > > >> >
>>> > > >>
>>> > >
>>> >
>>> https://issues.apache.org/jira/issues/?jql=project%20%3D%20FLINK%20AND%20status%20in%20(Open%2C%20%22In%20Progress%22%2C%20Reopened)%20AND%20affectedVersion%20in%20(1.11.0%2C%201.11.1%2C%201.11.2)%20AND%20labels%20%3D%20test-stability
>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>
>>> > > >> > > >>>>>>>>>>>>>>>>>>>>> On Mon, Oct 26, 2020 at 9:43 PM Kostas
>>> > > >> > > >>>>> Kloudas <
>>> > > >> > > >>>>>>>>>>>>>>>>> kkloudas@apache.org>
>>> > > >> > > >>>>>>>>>>>>>>>>>>>>> wrote:
>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>
>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>> +1 for releasing Flink 1.11.3 as it
>>> > > >> > > >>>>> contains a
>>> > > >> > > >>>>>>>>> number
>>> > > >> > > >>>>>>>>>>> of
>>> > > >> > > >>>>>>>>>>>>>>>>> important
>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>> fixes and thanks Gordon and Xintong
>>> > > >> > > >> for
>>> > > >> > > >>>>>>>>> volunteering.
>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>
>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>> Cheers,
>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>> Kostas
>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>
>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>> On Mon, Oct 26, 2020 at 4:37 AM Yu Li
>>> > > >> > > >> <
>>> > > >> > > >>>>>>>>>>> carp84@gmail.com>
>>> > > >> > > >>>>>>>>>>>>>>> wrote:
>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>
>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>> +1 for releasing Flink 1.11.3, and
>>> > > >> > > >>>> thanks
>>> > > >> > > >>>>>>> Gordon
>>> > > >> > > >>>>>>>>>> and
>>> > > >> > > >>>>>>>>>>>>>>> Xintong
>>> > > >> > > >>>>>>>>>>>>>>>>> for
>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>> volunteering as our release
>>> > > >> > > >> managers.
>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>
>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>> Best Regards,
>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>> Yu
>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>
>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>
>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>> On Mon, 26 Oct 2020 at 09:45,
>>> > > >> > > >> Xintong
>>> > > >> > > >>>>> Song <
>>> > > >> > > >>>>>>>>>>>>>>>>>> tonysong820@gmail.com>
>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>> wrote:
>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>
>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>> Thanks Gordan for starting this
>>> > > >> > > >>>>>> discussion.
>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>> My pleasure to help with the
>>> > > >> > > >> release
>>> > > >> > > >>>>>>> process.
>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>
>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>> Thank you~
>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>
>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>> Xintong Song
>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>
>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>
>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>
>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>> On Fri, Oct 23, 2020 at 11:29 PM
>>> > > >> > > >>> Till
>>> > > >> > > >>>>>>>>> Rohrmann <
>>> > > >> > > >>>>>>>>>>>>>>>>>>>> trohrmann@apache.org
>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>
>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>
>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>> Thanks for starting this
>>> > > >> > > >>> discussion
>>> > > >> > > >>>>>>> Gordon.
>>> > > >> > > >>>>>>>>>> There
>>> > > >> > > >>>>>>>>>>>> are
>>> > > >> > > >>>>>>>>>>>>>>> over
>>> > > >> > > >>>>>>>>>>>>>>>>>> 100
>>> > > >> > > >>>>>>>>>>>>>>>>>>>>> issues
>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>> which are fixed for 1.11.3.
>>> > > >> > > >> Hence
>>> > > >> > > >>> +1
>>> > > >> > > >>>>>> for a
>>> > > >> > > >>>>>>>>>>> soonish
>>> > > >> > > >>>>>>>>>>>>>>> 1.11.3
>>> > > >> > > >>>>>>>>>>>>>>>>>>>> release.
>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>> Thanks
>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>> for volunteering as our release
>>> > > >> > > >>>>> managers
>>> > > >> > > >>>>>>>>> Gordon
>>> > > >> > > >>>>>>>>>>> and
>>> > > >> > > >>>>>>>>>>>>>>>>> Xintong!
>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>
>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>> Cheers,
>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>> Till
>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>
>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>> On Fri, Oct 23, 2020 at 5:02 PM
>>> > > >> > > >>>> Tzu-Li
>>> > > >> > > >>>>>>>>> (Gordon)
>>> > > >> > > >>>>>>>>>>>> Tai <
>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>> tzulitai@apache.org
>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>
>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>
>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>> Hi,
>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>>
>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>> Xintong and I would like to
>>> > > >> > > >>> start a
>>> > > >> > > >>>>>>>>> discussion
>>> > > >> > > >>>>>>>>>>> for
>>> > > >> > > >>>>>>>>>>>>>>>>> releasing
>>> > > >> > > >>>>>>>>>>>>>>>>>>>> Flink
>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>> 1.11.3
>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>> soon.
>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>>
>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>> It seems like we already have a
>>> > > >> > > >>> few
>>> > > >> > > >>>>>>>>> pressing
>>> > > >> > > >>>>>>>>>>>> issues
>>> > > >> > > >>>>>>>>>>>>>>> that
>>> > > >> > > >>>>>>>>>>>>>>>>>> needs
>>> > > >> > > >>>>>>>>>>>>>>>>>>>> to
>>> > > >> > > >>>>>>>>>>>>>>>>>>>>> be
>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>> included in a new hotfix
>>> > > >> > > >> release:
>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>>
>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>>   - Heap-based timers’ restore
>>> > > >> > > >>>>>> behaviour
>>> > > >> > > >>>>>>>>> is
>>> > > >> > > >>>>>>>>>>>>> causing a
>>> > > >> > > >>>>>>>>>>>>>>>>>>> critical
>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>> recovery
>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>>   issue for StateFun [1] [2]
>>> > > >> > > >>> [3].
>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>>   - There are several
>>> > > >> > > >> robustness
>>> > > >> > > >>>>>> issues
>>> > > >> > > >>>>>>>>> for
>>> > > >> > > >>>>>>>>>> the
>>> > > >> > > >>>>>>>>>>>>>>> FLIP-27
>>> > > >> > > >>>>>>>>>>>>>>>>> new
>>> > > >> > > >>>>>>>>>>>>>>>>>>>>> source
>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>> API,
>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>>   such as [4]. We already have
>>> > > >> > > >>>> some
>>> > > >> > > >>>>>>> users
>>> > > >> > > >>>>>>>>>> using
>>> > > >> > > >>>>>>>>>>>> the
>>> > > >> > > >>>>>>>>>>>>>>>>> FLIP-27
>>> > > >> > > >>>>>>>>>>>>>>>>>>> API
>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>> with
>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>> 1.11.x,
>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>>   so it would be important to
>>> > > >> > > >>> get
>>> > > >> > > >>>>>> those
>>> > > >> > > >>>>>>>>> fixes
>>> > > >> > > >>>>>>>>>>> in
>>> > > >> > > >>>>>>>>>>>>> for
>>> > > >> > > >>>>>>>>>>>>>>>>> 1.11.x
>>> > > >> > > >>>>>>>>>>>>>>>>>>> as
>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>> well.
>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>>
>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>> Apart from the issues that are
>>> > > >> > > >>>>> already
>>> > > >> > > >>>>>>>>> marked
>>> > > >> > > >>>>>>>>>> as
>>> > > >> > > >>>>>>>>>>>>>>> blocker
>>> > > >> > > >>>>>>>>>>>>>>>>> for
>>> > > >> > > >>>>>>>>>>>>>>>>>>>>> 1.11.3
>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>> in
>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>> our
>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>> JIRA [5], please let us know in
>>> > > >> > > >>>> this
>>> > > >> > > >>>>>>>>> thread if
>>> > > >> > > >>>>>>>>>>>> there
>>> > > >> > > >>>>>>>>>>>>>>> is
>>> > > >> > > >>>>>>>>>>>>>>>>>>> already
>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>> ongoing
>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>> work for other important fixes
>>> > > >> > > >>> that
>>> > > >> > > >>>>> we
>>> > > >> > > >>>>>>>>> should
>>> > > >> > > >>>>>>>>>>> try
>>> > > >> > > >>>>>>>>>>>> to
>>> > > >> > > >>>>>>>>>>>>>>>>>> include.
>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>>
>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>> Xintong and I would like to
>>> > > >> > > >>>> volunteer
>>> > > >> > > >>>>>> for
>>> > > >> > > >>>>>>>>>>> managing
>>> > > >> > > >>>>>>>>>>>>>>> this
>>> > > >> > > >>>>>>>>>>>>>>>>>>> release,
>>> > > >> > > >>>>>>>>>>>>>>>>>>>>> and
>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>> will
>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>> try to communicate the priority
>>> > > >> > > >>> of
>>> > > >> > > >>>>>>> pending
>>> > > >> > > >>>>>>>>>>>> blockers
>>> > > >> > > >>>>>>>>>>>>>>> over
>>> > > >> > > >>>>>>>>>>>>>>>>> the
>>> > > >> > > >>>>>>>>>>>>>>>>>>>> next
>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>> few
>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>> days.
>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>> Since the aforementioned issues
>>> > > >> > > >>> are
>>> > > >> > > >>>>>> quite
>>> > > >> > > >>>>>>>>>>>> critical,
>>> > > >> > > >>>>>>>>>>>>>>> we’d
>>> > > >> > > >>>>>>>>>>>>>>>>>> like
>>> > > >> > > >>>>>>>>>>>>>>>>>>> to
>>> > > >> > > >>>>>>>>>>>>>>>>>>>>> aim
>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>> for a *feature
>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>> freeze by the end of next week
>>> > > >> > > >>>> (Oct.
>>> > > >> > > >>>>>>> 30th)*
>>> > > >> > > >>>>>>>>>> and
>>> > > >> > > >>>>>>>>>>>>> start
>>> > > >> > > >>>>>>>>>>>>>>> the
>>> > > >> > > >>>>>>>>>>>>>>>>>>>> release
>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>> voting
>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>> process the week after.
>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>> If that is too short of a
>>> > > >> > > >> notice
>>> > > >> > > >>>> and
>>> > > >> > > >>>>>> you
>>> > > >> > > >>>>>>>>> might
>>> > > >> > > >>>>>>>>>>>> need
>>> > > >> > > >>>>>>>>>>>>>>> more
>>> > > >> > > >>>>>>>>>>>>>>>>>> time,
>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>> please
>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>> let
>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>> us know!
>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>>
>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>> Cheers,
>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>> Gordon
>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>>
>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>> [1]
>>> > > >> > > >>>>>>>>>>>>>
>>> https://issues.apache.org/jira/browse/FLINK-19692
>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>> [2]
>>> > > >> > > >>>>>>>>>>>>>
>>> https://issues.apache.org/jira/browse/FLINK-19741
>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>> [3]
>>> > > >> > > >>>>>>>>>>>>>
>>> https://issues.apache.org/jira/browse/FLINK-19748
>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>> [4]
>>> > > >> > > >>>>>>>>>>>>>
>>> https://issues.apache.org/jira/browse/FLINK-19717
>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>> [5]
>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>>
>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>>
>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>
>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>
>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>
>>> > > >> > > >>>>>>>>>>>>>>>>>>>>
>>> > > >> > > >>>>>>>>>>>>>>>>>>>
>>> > > >> > > >>>>>>>>>>>>>>>>>>
>>> > > >> > > >>>>>>>>>>>>>>>>>
>>> > > >> > > >>>>>>>>>>>>>>>
>>> > > >> > > >>>>>>>>>>>>>
>>> > > >> > > >>>>>>>>>>>>
>>> > > >> > > >>>>>>>>>>>
>>> > > >> > > >>>>>>>>>>
>>> > > >> > > >>>>>>>>>
>>> > > >> > > >>>>>>>
>>> > > >> > > >>>>>>
>>> > > >> > > >>>>>
>>> > > >> > > >>>>
>>> > > >> > > >>>
>>> > > >> > > >>
>>> > > >> > >
>>> > > >> >
>>> > > >>
>>> > >
>>> >
>>> https://issues.apache.org/jira/issues/?jql=project%20%3D%20FLINK%20AND%20priority%20%3D%20Blocker%20AND%20fixVersion%20%3D%201.11.3
>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>>
>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>
>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>
>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>
>>> > > >> > > >>>>>>>>>>>>>>>>>>>>>
>>> > > >> > > >>>>>>>>>>>>>>>>>>>>
>>> > > >> > > >>>>>>>>>>>>>>>>>>>
>>> > > >> > > >>>>>>>>>>>>>>>>>>
>>> > > >> > > >>>>>>>>>>>>>>>>>>
>>> > > >> > > >>>>>>>>>>>>>>>>>> --
>>> > > >> > > >>>>>>>>>>>>>>>>>> Best, Jingsong Lee
>>> > > >> > > >>>>>>>>>>>>>>>>>>
>>> > > >> > > >>>>>>>>>>>>>>>>>
>>> > > >> > > >>>>>>>>>>>>>>>>
>>> > > >> > > >>>>>>>>>>>>>>>
>>> > > >> > > >>>>>>>>>>>>>>
>>> > > >> > > >>>>>>>>>>>>>
>>> > > >> > > >>>>>>>>>>>>
>>> > > >> > > >>>>>>>>>>>
>>> > > >> > > >>>>>>>>>>
>>> > > >> > > >>>>>>>>>
>>> > > >> > > >>>>>>>>
>>> > > >> > > >>>>>>>
>>> > > >> > > >>>>>>
>>> > > >> > > >>>>>
>>> > > >> > > >>>>
>>> > > >> > > >>>
>>> > > >> > > >>
>>> > > >> > >
>>> > > >> > >
>>> > > >> > >
>>> > > >> >
>>> > > >>
>>> > > >
>>> > >
>>> >
>>>
>>

Re: [DISCUSS] Releasing Apache Flink 1.11.3

Posted by Xintong Song <to...@gmail.com>.
Hi devs,

Updates on the release progress.

There are a couple of new blockers emerged since the last update, all of
which have been resolved in the past week.

The release 1.11.3 is still blocked on FLIP-27.
Stephan and Becket are trying to first get the feature stabilized on the
master branch for release 1.12. After that, the remaining changes will be
backported to 1.11, and the release 1.11.3 will be unblocked.

We are hoping to deliver the first RC by the end of next week. During this
time, if there's any other things that you believe should be included in
this release, please reach out to either me or Gordon.

Sorry for the delay.

Thank you~

Xintong Song



On Mon, Nov 23, 2020 at 2:03 PM Xintong Song <to...@gmail.com> wrote:

> Thanks for the notice, Zhu.
> Let's have the discussion on the jira ticket.
>
> Thank you~
>
> Xintong Song
>
>
>
> On Mon, Nov 23, 2020 at 12:10 PM Zhu Zhu <re...@gmail.com> wrote:
>
>> Hi Xintong,
>> We just found an issue FLINK-20285 and I think it is a blocker for 1.11.3.
>>
>> Thanks,
>> Zhu
>>
>> Gyula Fóra <gy...@gmail.com> 于2020年11月18日周三 下午9:36写道：
>>
>> > Hi All!
>> >
>> > I have found the following issue today which might be considered a
>> blocker
>> > for this release as well:
>> >
>> > https://issues.apache.org/jira/browse/FLINK-20221
>> >
>> > Could someone please quickly provide a second set of eyes and confirm
>> that
>> > this is indeed a big problem? :)
>> >
>> > Thank you!
>> > Gyula
>> >
>> > On Wed, Nov 18, 2020 at 5:12 AM Xintong Song <to...@gmail.com>
>> > wrote:
>> >
>> > > Hi devs,
>> > >
>> > > Updates on the progress of preparing the 1.11.3 release.
>> > >
>> > > We are approaching the creation of our first release candidate. Thanks
>> > all
>> > > for the efforts so far.
>> > >
>> > > # Ongoing threads
>> > >
>> > >    - *FLIP-27 backportings:* 74 out of 102 commits are already
>> > backported.
>> > >    Stephan and Becket are actively working on this.
>> > >    - *Blockers:* There are 2 remaining blockers, both have been fixed
>> on
>> > >    the master branch and should be backported along with the FLIP-27
>> > > changes.
>> > >
>> > > As soon as the above threads are done, we will create our first
>> release
>> > > candidate. If there's any other issues that you believe should be a
>> > release
>> > > blocker, please reach out to either me or Gordon.
>> > >
>> > > # JIRA version
>> > >
>> > > Version 1.11.4 has been created on JIRA. I've already tagged some of
>> the
>> > > tickets with the new fix version. Please feel free to continue with
>> the
>> > > efforts and merge once they are ready. I'll double check and update
>> the
>> > fix
>> > > versions before the release.
>> > >
>> > > Thank you~
>> > >
>> > > Xintong Song
>> > >
>> > >
>> > >
>> > > On Thu, Nov 12, 2020 at 1:31 PM Xintong Song <to...@gmail.com>
>> > > wrote:
>> > >
>> > > > Thanks for the notice and fix, Roman.
>> > > >
>> > > > Thank you~
>> > > >
>> > > > Xintong Song
>> > > >
>> > > >
>> > > >
>> > > > On Wed, Nov 11, 2020 at 5:53 PM Khachatryan Roman <
>> > > > khachatryan.roman@gmail.com> wrote:
>> > > >
>> > > >> Hi,
>> > > >>
>> > > >> I'd like FLINK-20079 [1] to be merged into 1.11 and included in
>> > 1.11.3.
>> > > >>
>> > > >> [1] https://issues.apache.org/jira/browse/FLINK-20079
>> > > >>
>> > > >> Regards,
>> > > >> Roman
>> > > >>
>> > > >>
>> > > >> On Tue, Nov 10, 2020 at 8:21 AM Xintong Song <
>> tonysong820@gmail.com>
>> > > >> wrote:
>> > > >>
>> > > >> > Thanks for the notice, Dian.
>> > > >> >
>> > > >> > Thank you~
>> > > >> >
>> > > >> > Xintong Song
>> > > >> >
>> > > >> >
>> > > >> >
>> > > >> > On Tue, Nov 10, 2020 at 10:19 AM Dian Fu <di...@gmail.com>
>> > > wrote:
>> > > >> >
>> > > >> > > Hi Xintong,
>> > > >> > >
>> > > >> > > I want to bring one more issue to your attention [1]. The test
>> > case
>> > > >> > > UnalignedCheckpointCompatibilityITCase.test failed several
>> times
>> > in
>> > > >> the
>> > > >> > > last nightly test of release-1.11. We need to figure out if
>> it's
>> > > just
>> > > >> an
>> > > >> > > instable test or caused by recent changes.
>> > > >> > >
>> > > >> > > [1] https://issues.apache.org/jira/browse/FLINK-20065
>> > > >> > >
>> > > >> > > > 在 2020年11月10日，上午9:24，Xintong Song <to...@gmail.com>
>> 写道：
>> > > >> > > >
>> > > >> > > > Thanks for the replies.
>> > > >> > > >
>> > > >> > > > Thank you~
>> > > >> > > >
>> > > >> > > > Xintong Song
>> > > >> > > >
>> > > >> > > >
>> > > >> > > >
>> > > >> > > > On Tue, Nov 10, 2020 at 1:09 AM Becket Qin <
>> > becket.qin@gmail.com>
>> > > >> > wrote:
>> > > >> > > >
>> > > >> > > >> Hi Xintong,
>> > > >> > > >>
>> > > >> > > >> Thanks for driving the release. Just want to sync up on the
>> > > FLIP-27
>> > > >> > > >> backporting. Stephan and I are still trying to backport a
>> bunch
>> > > of
>> > > >> > > patches
>> > > >> > > >> of Source to 1.11.3. Including:
>> > > >> > > >>
>> > > >> > > >> [FLINK-19698][connector/common] Add a close() method to the
>> > > >> > SplitReader.
>> > > >> > > >> [FLINK-19717] SourceReaderBase.pollNext may return
>> END_OF_INPUT
>> > > if
>> > > >> > > >> SplitReader.fetch throws
>> > > >> > > >> [FLINK-19535] [connector/common] Avoid failing a job
>> multiple
>> > > >> times in
>> > > >> > > >> SourceCoordinator.
>> > > >> > > >> [FLINK-19265] [FLINK-20049][core] Source API final
>> adjustments.
>> > > >> > > >>
>> > > >> > > >> and a few more fixes.
>> > > >> > > >>
>> > > >> > > >> We are currently trying to fix them in 1.12 first so it
>> might
>> > > take
>> > > >> a
>> > > >> > > little
>> > > >> > > >> longer to backport them to 1.11.3. I think it will probably
>> > take
>> > > >> us a
>> > > >> > > few
>> > > >> > > >> more days to finish the backport. So that would roughly be
>> the
>> > > end
>> > > >> of
>> > > >> > > this
>> > > >> > > >> week.
>> > > >> > > >>
>> > > >> > > >> Thanks,
>> > > >> > > >>
>> > > >> > > >> Jiangjie (Becket) Qin
>> > > >> > > >>
>> > > >> > > >>
>> > > >> > > >>
>> > > >> > > >>
>> > > >> > > >> On Mon, Nov 9, 2020 at 9:57 PM Till Rohrmann <
>> > > trohrmann@apache.org
>> > > >> >
>> > > >> > > wrote:
>> > > >> > > >>
>> > > >> > > >>> Yes, I've downgraded FLINK-19816 to critical.
>> > > >> > > >>>
>> > > >> > > >>> Cheers,
>> > > >> > > >>> Till
>> > > >> > > >>>
>> > > >> > > >>> On Mon, Nov 9, 2020 at 10:19 AM Xintong Song <
>> > > >> tonysong820@gmail.com>
>> > > >> > > >>> wrote:
>> > > >> > > >>>
>> > > >> > > >>>> Thanks for the notice, Till.
>> > > >> > > >>>>
>> > > >> > > >>>> I just checked and found FLINK-20033 is already fixed.
>> Shall
>> > we
>> > > >> also
>> > > >> > > >>>> downgrade FLINK-19816 to `Critical`?
>> > > >> > > >>>>
>> > > >> > > >>>> Thank you~
>> > > >> > > >>>>
>> > > >> > > >>>> Xintong Song
>> > > >> > > >>>>
>> > > >> > > >>>>
>> > > >> > > >>>>
>> > > >> > > >>>> On Mon, Nov 9, 2020 at 4:42 PM Till Rohrmann <
>> > > >> trohrmann@apache.org>
>> > > >> > > >>> wrote:
>> > > >> > > >>>>
>> > > >> > > >>>>> I would like to bring one more critical issue to your
>> > > attention
>> > > >> > which
>> > > >> > > >>> is
>> > > >> > > >>>>> FLINK-20033 [1]. I believe that this issue is actually
>> > causing
>> > > >> what
>> > > >> > > >> has
>> > > >> > > >>>>> been reported in FLINK-19816 [2]. I hope to have it
>> fixed by
>> > > the
>> > > >> > end
>> > > >> > > >> of
>> > > >> > > >>>>> today. Once FLINK-20033 is fixed, I think that we don't
>> have
>> > > to
>> > > >> > block
>> > > >> > > >>> the
>> > > >> > > >>>>> release on FLINK-19816.
>> > > >> > > >>>>>
>> > > >> > > >>>>> [1] https://issues.apache.org/jira/browse/FLINK-20033
>> > > >> > > >>>>> [2] https://issues.apache.org/jira/browse/FLINK-19816
>> > > >> > > >>>>>
>> > > >> > > >>>>> Cheers,
>> > > >> > > >>>>> Till
>> > > >> > > >>>>>
>> > > >> > > >>>>> On Mon, Nov 9, 2020 at 4:05 AM Xintong Song <
>> > > >> tonysong820@gmail.com
>> > > >> > >
>> > > >> > > >>>> wrote:
>> > > >> > > >>>>>
>> > > >> > > >>>>>> Hi devs,
>> > > >> > > >>>>>>
>> > > >> > > >>>>>> I'd like to provide an update on the progress of
>> preparing
>> > > >> release
>> > > >> > > >>>>> 1.11.3.
>> > > >> > > >>>>>>
>> > > >> > > >>>>>> *Blockers*
>> > > >> > > >>>>>> We currently have 3 remaining blockers. (3 resolved and
>> 1
>> > > >> emerged
>> > > >> > > >>>>> compared
>> > > >> > > >>>>>> to last week)
>> > > >> > > >>>>>>
>> > > >> > > >>>>>> - [FLINK-19698] Add close() method and
>> > onCheckpointComplete()
>> > > >> to
>> > > >> > > >> the
>> > > >> > > >>>>>> Source.
>> > > >> > > >>>>>> The issue has been fixed on the master branch. It's
>> > currently
>> > > >> > > >> blocked
>> > > >> > > >>>> on
>> > > >> > > >>>>>> the FLIP-27 backportings to backport it to the 1.11
>> branch.
>> > > >> > > >>>>>>
>> > > >> > > >>>>>> - [FLINK-19717] SourceReaderBase.pollNext may return
>> > > >> END_OF_INPUT
>> > > >> > > >> if
>> > > >> > > >>>>>> SplitReader.fetch throws
>> > > >> > > >>>>>> A PR has been opened and reviewed. From the discussions
>> on
>> > > the
>> > > >> PR,
>> > > >> > > >> it
>> > > >> > > >>>>> looks
>> > > >> > > >>>>>> close to mergeable.
>> > > >> > > >>>>>>
>> > > >> > > >>>>>> - [FLINK-19816] Flink restored from a wrong checkpoint
>> (a
>> > > very
>> > > >> old
>> > > >> > > >>> one
>> > > >> > > >>>>> and
>> > > >> > > >>>>>> not the last completed one)
>> > > >> > > >>>>>> This is a newly emerged blocker and Matthias is working
>> on
>> > > it.
>> > > >> > > >>>>>>
>> > > >> > > >>>>>> *Test Instabilities*
>> > > >> > > >>>>>> We currently have 27 test instabilities[1].
>> > > >> > > >>>>>> AFAIK, none of them are as serious as to block the
>> 1.11.3
>> > > >> release.
>> > > >> > > >>>>>>
>> > > >> > > >>>>>> *FLIP-27 Backprotings*
>> > > >> > > >>>>>>
>> > > >> > > >>>>>> I noticed that there's no jira issues opened on the
>> FLIP-27
>> > > >> > > >>> backporting
>> > > >> > > >>>>>> efforts, which is part of the major efforts planned for
>> the
>> > > >> 1.11.3
>> > > >> > > >>>>> release,
>> > > >> > > >>>>>> making it hard to track the progress.
>> > > >> > > >>>>>>
>> > > >> > > >>>>>>
>> > > >> > > >>>>>> @Stephan and @Becket, could you please share the
>> updates on
>> > > the
>> > > >> > > >>>>> backporting
>> > > >> > > >>>>>> efforts? How is the progress and when are the efforts
>> > > expected
>> > > >> to
>> > > >> > > >> be
>> > > >> > > >>>>>> finished? It would be appreciated and helpful if we can
>> > have
>> > > a
>> > > >> > jira
>> > > >> > > >>>>> ticket
>> > > >> > > >>>>>> to track the backporing progress publicly.
>> > > >> > > >>>>>>
>> > > >> > > >>>>>>
>> > > >> > > >>>>>> Thank you~
>> > > >> > > >>>>>>
>> > > >> > > >>>>>> Xintong Song
>> > > >> > > >>>>>>
>> > > >> > > >>>>>>
>> > > >> > > >>>>>> [1]
>> > > >> > > >>>>>>
>> > > >> > > >>>>>>
>> > > >> > > >>>>>
>> > > >> > > >>>>
>> > > >> > > >>>
>> > > >> > > >>
>> > > >> > >
>> > > >> >
>> > > >>
>> > >
>> >
>> https://issues.apache.org/jira/browse/FLINK-16908?filter=12348580&jql=project%20%3D%20FLINK%20AND%20resolution%20%3D%20Unresolved%20AND%20affectedVersion%20in%20(1.11.0%2C%201.11.1%2C%201.11.2%2C%201.11.3)%20AND%20labels%20%3D%20test-stability%20ORDER%20BY%20updated%20DESC%2C%20created%20DESC
>> > > >> > > >>>>>>
>> > > >> > > >>>>>> On Wed, Nov 4, 2020 at 3:58 PM Tzu-Li (Gordon) Tai <
>> > > >> > > >>>> tzulitai@apache.org>
>> > > >> > > >>>>>> wrote:
>> > > >> > > >>>>>>
>> > > >> > > >>>>>>> Thanks for the efforts so far with pushing for 1.11.3.
>> > > >> > > >>>>>>>
>> > > >> > > >>>>>>> I'd like to provide a mid-week update on how we're
>> looking
>> > > >> with
>> > > >> > > >> all
>> > > >> > > >>>> the
>> > > >> > > >>>>>>> pending blockers and backports:
>> > > >> > > >>>>>>>
>> > > >> > > >>>>>>> *Blockers**:*
>> > > >> > > >>>>>>>
>> > > >> > > >>>>>>> Currently all blockers either have an open PR, or have
>> > been
>> > > >> > > >> merged.
>> > > >> > > >>>>> I'll
>> > > >> > > >>>>>>> highlight below the blockers *that still require
>> reviewing
>> > > >> > > >> efforts
>> > > >> > > >>>> for
>> > > >> > > >>>>>> them
>> > > >> > > >>>>>>> to move forward:*
>> > > >> > > >>>>>>>
>> > > >> > > >>>>>>> - [FLINK-19909] Flink application in attach mode could
>> not
>> > > >> > > >>> terminate
>> > > >> > > >>>>> when
>> > > >> > > >>>>>>> the only job is canceled. PR:
>> > > >> > > >>>>> https://github.com/apache/flink/pull/13911
>> > > >> > > >>>>>>> - [FLINK-19717] SourceReaderBase.pollNext may return
>> > > >> END_OF_INPUT
>> > > >> > > >>> if
>> > > >> > > >>>>>>> SplitReader.fetch throws. PR:
>> > > >> > > >>>>> https://github.com/apache/flink/pull/13776
>> > > >> > > >>>>>>>
>> > > >> > > >>>>>>> The above PRs currently have no reviews at all yet.
>> They
>> > > seem
>> > > >> to
>> > > >> > > >>>>> already
>> > > >> > > >>>>>>> have designated reviewers.
>> > > >> > > >>>>>>>
>> > > >> > > >>>>>>>
>> > > >> > > >>>>>>> *Backports:*
>> > > >> > > >>>>>>> - FLIP-27 robustness improvement backports: Stephan is
>> > > >> currently
>> > > >> > > >>>>> working
>> > > >> > > >>>>>> on
>> > > >> > > >>>>>>> backporting several FLIP-27 changes. There are no PRs
>> yet
>> > > for
>> > > >> the
>> > > >> > > >>>>>>> backported changes.
>> > > >> > > >>>>>>> - DataStreamUtils.collect() refactorings backport:
>> Steven
>> > Wu
>> > > >> > > >>>> mentioned
>> > > >> > > >>>>> to
>> > > >> > > >>>>>>> backport this, but AFAIK this isn't assigned to anyone
>> > yet.
>> > > >> > > >>>>>>>
>> > > >> > > >>>>>>> *ETAs*:
>> > > >> > > >>>>>>>
>> > > >> > > >>>>>>> I'd like to request ETAs for the remaining backports,
>> to
>> > > >> prevent
>> > > >> > > >> a
>> > > >> > > >>>>> creep
>> > > >> > > >>>>>> in
>> > > >> > > >>>>>>> the scope of this bugfix release.
>> > > >> > > >>>>>>>
>> > > >> > > >>>>>>> We already have Flink users that would benefit from
>> fixes
>> > > that
>> > > >> > > >> have
>> > > >> > > >>>>> been
>> > > >> > > >>>>>>> merged to release-1.11 already, so technically speaking
>> > the
>> > > >> > > >>> backports
>> > > >> > > >>>>>>> should be considered "nice-to-have" (to the best of my
>> > > >> knowledge
>> > > >> > > >> of
>> > > >> > > >>>> the
>> > > >> > > >>>>>>> changes) and could potentially be moved to a follow-up
>> > > 1.11.4.
>> > > >> > > >>>>>>> Most notably, the Stateful Functions project is already
>> > > >> waiting
>> > > >> > > >> on
>> > > >> > > >>>>> Flink
>> > > >> > > >>>>>>> 1.11.3 to address critical recovery issues (please see
>> the
>> > > >> > > >> StateFun
>> > > >> > > >>>>> 2.2.1
>> > > >> > > >>>>>>> release discussion thread [1]).
>> > > >> > > >>>>>>>
>> > > >> > > >>>>>>> @Stephan Ewen <se...@apache.org> @Becket Qin <
>> > > >> > > >> becket.qin@gmail.com
>> > > >> > > >>>>
>> > > >> > > >>>>>> could
>> > > >> > > >>>>>>> you provide an ETA for the FLIP-27 backports? It would
>> > help
>> > > to
>> > > >> > > >> get
>> > > >> > > >>> a
>> > > >> > > >>>>>> better
>> > > >> > > >>>>>>> estimate to decide how we proceed here.
>> > > >> > > >>>>>>>
>> > > >> > > >>>>>>> Cheers,
>> > > >> > > >>>>>>> Gordon
>> > > >> > > >>>>>>>
>> > > >> > > >>>>>>> [1]
>> > > >> > > >>>>>>>
>> > > >> > > >>>>>>>
>> > > >> > > >>>>>>
>> > > >> > > >>>>>
>> > > >> > > >>>>
>> > > >> > > >>>
>> > > >> > > >>
>> > > >> > >
>> > > >> >
>> > > >>
>> > >
>> >
>> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Releasing-StateFun-hotfix-version-2-2-1-td46239.html
>> > > >> > > >>>>>>>
>> > > >> > > >>>>>>> On Wed, Nov 4, 2020 at 3:16 PM Tzu-Li (Gordon) Tai <
>> > > >> > > >>>>> tzulitai@apache.org>
>> > > >> > > >>>>>>> wrote:
>> > > >> > > >>>>>>>
>> > > >> > > >>>>>>>>> The collect() utils can be picked back, I see no
>> issue
>> > > with
>> > > >> > > >>> that
>> > > >> > > >>>>> (it
>> > > >> > > >>>>>> is
>> > > >> > > >>>>>>>> isolated utilities).
>> > > >> > > >>>>>>>>
>> > > >> > > >>>>>>>> Just checking on all the requested backports
>> mentioned in
>> > > >> this
>> > > >> > > >>>>> thread,
>> > > >> > > >>>>>>> and
>> > > >> > > >>>>>>>> figuring out which ones seem to still be unassigned /
>> > open.
>> > > >> > > >>>>>>>>
>> > > >> > > >>>>>>>> Is someone working on backporting
>> > > >> > > >>>>>>>>
>> > > >> > > >>>>>>>
>> > > >> > > >>>>>>
>> > > >> > > >>>>>
>> > > >> > > >>>>
>> > > >> > > >>>
>> > > >> > > >>
>> > > >> > >
>> > > >> >
>> > > >>
>> > >
>> >
>> https://github.com/apache/flink/commit/09a7a66b7313fea64817fe960a8da1265b428efc
>> > > >> > > >>>>>>>> to release-1.11 at the moment?
>> > > >> > > >>>>>>>>
>> > > >> > > >>>>>>>> On Wed, Nov 4, 2020 at 1:39 AM Steven Wu <
>> > > >> stevenz3wu@gmail.com
>> > > >> > > >>>
>> > > >> > > >>>>> wrote:
>> > > >> > > >>>>>>>>
>> > > >> > > >>>>>>>>> @Stephan Ewen <se...@apache.org> yeah, we can do
>> that.
>> > > >> don't
>> > > >> > > >>>> worry
>> > > >> > > >>>>>>> about
>> > > >> > > >>>>>>>>> it. your earlier email had the perfect explanation on
>> > why
>> > > >> file
>> > > >> > > >>>>> source
>> > > >> > > >>>>>>>>> shouldn't be backported.
>> > > >> > > >>>>>>>>>
>> > > >> > > >>>>>>>>> On Tue, Nov 3, 2020 at 3:37 AM Stephan Ewen <
>> > > >> sewen@apache.org
>> > > >> > > >>>
>> > > >> > > >>>>> wrote:
>> > > >> > > >>>>>>>>>
>> > > >> > > >>>>>>>>>> @Steven would it be possible to initially copy some
>> of
>> > > the
>> > > >> > > >>> code
>> > > >> > > >>>>> into
>> > > >> > > >>>>>>> the
>> > > >> > > >>>>>>>>>> iceberg source and later replace it by a dependency
>> on
>> > > the
>> > > >> > > >>> Flink
>> > > >> > > >>>>>> file
>> > > >> > > >>>>>>>>>> source?
>> > > >> > > >>>>>>>>>>
>> > > >> > > >>>>>>>>>> On Mon, Nov 2, 2020 at 8:33 PM Steven Wu <
>> > > >> > > >>> stevenz3wu@gmail.com>
>> > > >> > > >>>>>>> wrote:
>> > > >> > > >>>>>>>>>>
>> > > >> > > >>>>>>>>>>> Stephan, thanks a lot for explaining the file
>> > connector.
>> > > >> > > >>> that
>> > > >> > > >>>>>> makes
>> > > >> > > >>>>>>>>>> sense.
>> > > >> > > >>>>>>>>>>>
>> > > >> > > >>>>>>>>>>> I was asking because we were trying to reuse some
>> of
>> > the
>> > > >> > > >>>>>>>>> implementations
>> > > >> > > >>>>>>>>>> in
>> > > >> > > >>>>>>>>>>> the file source for Iceberg source. Flink Iceberg
>> > source
>> > > >> > > >>> lives
>> > > >> > > >>>>> in
>> > > >> > > >>>>>>> the
>> > > >> > > >>>>>>>>>>> Iceberg repo, which is not possible to code against
>> > the
>> > > >> > > >>> master
>> > > >> > > >>>>>>> branch
>> > > >> > > >>>>>>>>> of
>> > > >> > > >>>>>>>>>>> the Flink code.
>> > > >> > > >>>>>>>>>>>
>> > > >> > > >>>>>>>>>>> On Mon, Nov 2, 2020 at 3:31 AM Stephan Ewen <
>> > > >> > > >>> sewen@apache.org
>> > > >> > > >>>>>
>> > > >> > > >>>>>>> wrote:
>> > > >> > > >>>>>>>>>>>
>> > > >> > > >>>>>>>>>>>> Hi Steven!
>> > > >> > > >>>>>>>>>>>>
>> > > >> > > >>>>>>>>>>>> So far there are no plans to pick back the file
>> > system
>> > > >> > > >>>>> connector
>> > > >> > > >>>>>>>>> code.
>> > > >> > > >>>>>>>>>>> This
>> > > >> > > >>>>>>>>>>>> is still evolving and not finalized for 1.12, so I
>> > > don't
>> > > >> > > >>>> feel
>> > > >> > > >>>>> it
>> > > >> > > >>>>>>> is
>> > > >> > > >>>>>>>>> a
>> > > >> > > >>>>>>>>>>> good
>> > > >> > > >>>>>>>>>>>> candidate to be backported.
>> > > >> > > >>>>>>>>>>>> However, with the base connector changes
>> backported,
>> > > you
>> > > >> > > >>>>> should
>> > > >> > > >>>>>> be
>> > > >> > > >>>>>>>>> able
>> > > >> > > >>>>>>>>>>> to
>> > > >> > > >>>>>>>>>>>> run the file connector code from master against
>> > 1.11.3.
>> > > >> > > >>>>>>>>>>>>
>> > > >> > > >>>>>>>>>>>> The collect() utils can be picked back, I see no
>> > issue
>> > > >> > > >>> with
>> > > >> > > >>>>> that
>> > > >> > > >>>>>>>>> (it is
>> > > >> > > >>>>>>>>>>>> isolated utilities).
>> > > >> > > >>>>>>>>>>>>
>> > > >> > > >>>>>>>>>>>> Best,
>> > > >> > > >>>>>>>>>>>> Stephan
>> > > >> > > >>>>>>>>>>>>
>> > > >> > > >>>>>>>>>>>>
>> > > >> > > >>>>>>>>>>>> On Mon, Nov 2, 2020 at 3:02 AM Steven Wu <
>> > > >> > > >>>>> stevenz3wu@gmail.com>
>> > > >> > > >>>>>>>>> wrote:
>> > > >> > > >>>>>>>>>>>>
>> > > >> > > >>>>>>>>>>>>> Basically, it would be great to get the latest
>> code
>> > in
>> > > >> > > >>> the
>> > > >> > > >>>>>>>>>>>>> flink-connector-files (FLIP-27).
>> > > >> > > >>>>>>>>>>>>>
>> > > >> > > >>>>>>>>>>>>> On Sat, Oct 31, 2020 at 9:57 AM Steven Wu <
>> > > >> > > >>>>>> stevenz3wu@gmail.com
>> > > >> > > >>>>>>>>
>> > > >> > > >>>>>>>>>>> wrote:
>> > > >> > > >>>>>>>>>>>>>
>> > > >> > > >>>>>>>>>>>>>> Stephan, it will be great if we can also
>> backport
>> > > >> > > >> the
>> > > >> > > >>>>>>>>>> DataStreamUtils
>> > > >> > > >>>>>>>>>>>>>> related commits that help with collecting output
>> > > >> > > >> from
>> > > >> > > >>>>>>> unbounded
>> > > >> > > >>>>>>>>>>>> streams.
>> > > >> > > >>>>>>>>>>>>>> e.g.
>> > > >> > > >>>>>>>>>>>>>>
>> > > >> > > >>>>>>>>>>>>>>
>> > > >> > > >>>>>>>>>>>>>>
>> > > >> > > >>>>>>>>>>>>>
>> > > >> > > >>>>>>>>>>>>
>> > > >> > > >>>>>>>>>>>
>> > > >> > > >>>>>>>>>>
>> > > >> > > >>>>>>>>>
>> > > >> > > >>>>>>>
>> > > >> > > >>>>>>
>> > > >> > > >>>>>
>> > > >> > > >>>>
>> > > >> > > >>>
>> > > >> > > >>
>> > > >> > >
>> > > >> >
>> > > >>
>> > >
>> >
>> https://github.com/apache/flink/commit/09a7a66b7313fea64817fe960a8da1265b428efc
>> > > >> > > >>>>>>>>>>>>>>
>> > > >> > > >>>>>>>>>>>>>> I tried to copy and paste the code to unblock
>> > > >> > > >> myself.
>> > > >> > > >>>> but
>> > > >> > > >>>>> it
>> > > >> > > >>>>>>>>>> quickly
>> > > >> > > >>>>>>>>>>>> got
>> > > >> > > >>>>>>>>>>>>>> into the rabbit hole of more and more code.
>> > > >> > > >>>>>>>>>>>>>>
>> > > >> > > >>>>>>>>>>>>>> On Fri, Oct 30, 2020 at 11:02 AM Stephan Ewen <
>> > > >> > > >>>>>>> sewen@apache.org
>> > > >> > > >>>>>>>>>>
>> > > >> > > >>>>>>>>>>>> wrote:
>> > > >> > > >>>>>>>>>>>>>>
>> > > >> > > >>>>>>>>>>>>>>> I have started with backporting the source API
>> > > >> > > >>> changes.
>> > > >> > > >>>>>> Some
>> > > >> > > >>>>>>>>> minor
>> > > >> > > >>>>>>>>>>>>>>> conflicts to solve, will need a bit more to
>> finish
>> > > >> > > >>>> this.
>> > > >> > > >>>>>>>>>>>>>>>
>> > > >> > > >>>>>>>>>>>>>>> On Fri, Oct 30, 2020 at 7:25 AM Tzu-Li (Gordon)
>> > > >> > > >> Tai <
>> > > >> > > >>>>>>>>>>>>> tzulitai@apache.org>
>> > > >> > > >>>>>>>>>>>>>>> wrote:
>> > > >> > > >>>>>>>>>>>>>>>
>> > > >> > > >>>>>>>>>>>>>>>> @Stephan Ewen <se...@apache.org>
>> > > >> > > >>>>>>>>>>>>>>>> Are there already plans or ongoing efforts for
>> > > >> > > >>>>>> backporting
>> > > >> > > >>>>>>>>> the
>> > > >> > > >>>>>>>>>>> list
>> > > >> > > >>>>>>>>>>>> of
>> > > >> > > >>>>>>>>>>>>>>>> FLIP-27 changes that you posted?
>> > > >> > > >>>>>>>>>>>>>>>>
>> > > >> > > >>>>>>>>>>>>>>>> On Thu, Oct 29, 2020 at 7:08 PM Xintong Song <
>> > > >> > > >>>>>>>>>>> tonysong820@gmail.com
>> > > >> > > >>>>>>>>>>>>>
>> > > >> > > >>>>>>>>>>>>>>>> wrote:
>> > > >> > > >>>>>>>>>>>>>>>>
>> > > >> > > >>>>>>>>>>>>>>>>> Hi folks,
>> > > >> > > >>>>>>>>>>>>>>>>>
>> > > >> > > >>>>>>>>>>>>>>>>> Just to provide some updates concerning the
>> > > >> > > >> status
>> > > >> > > >>>> on
>> > > >> > > >>>>>> the
>> > > >> > > >>>>>>>>>>>>>>>>> test instabilities.
>> > > >> > > >>>>>>>>>>>>>>>>>
>> > > >> > > >>>>>>>>>>>>>>>>> Currently, we have 30 unresolved tickets
>> labeled
>> > > >> > > >>>> with
>> > > >> > > >>>>>>>>> `Affects
>> > > >> > > >>>>>>>>>>>>> Version`
>> > > >> > > >>>>>>>>>>>>>>>>> 1.11.x.
>> > > >> > > >>>>>>>>>>>>>>>>>
>> > > >> > > >>>>>>>>>>>>>>>>>
>> > > >> > > >>>>>>>>>>>>>>>
>> > > >> > > >>>>>>>>>>>>>
>> > > >> > > >>>>>>>>>>>>
>> > > >> > > >>>>>>>>>>>
>> > > >> > > >>>>>>>>>>
>> > > >> > > >>>>>>>>>
>> > > >> > > >>>>>>>
>> > > >> > > >>>>>>
>> > > >> > > >>>>>
>> > > >> > > >>>>
>> > > >> > > >>>
>> > > >> > > >>
>> > > >> > >
>> > > >> >
>> > > >>
>> > >
>> >
>> https://issues.apache.org/jira/browse/FLINK-19775?filter=12348580&jql=project%20%3D%20FLINK%20AND%20resolution%20%3D%20Unresolved%20AND%20affectedVersion%20in%20(1.11.0%2C%201.11.1%2C%201.11.2%2C%201.11.3)%20AND%20labels%20%3D%20test-stability%20ORDER%20BY%20created%20DESC
>> > > >> > > >>>>>>>>>>>>>>>>>
>> > > >> > > >>>>>>>>>>>>>>>>> Among the 30 tickets, 11 of them are:
>> > > >> > > >>>>>>>>>>>>>>>>> - Have occured in the recent 3 months
>> > > >> > > >>>>>>>>>>>>>>>>> - Not confirmed to be pure testability issues
>> > > >> > > >>>>>>>>>>>>>>>>> - Not confirmed to be rare condition cases
>> > > >> > > >>>>>>>>>>>>>>>>>
>> > > >> > > >>>>>>>>>>>>>>>>> It would be nice if someone familiar with
>> these
>> > > >> > > >>>>>> components
>> > > >> > > >>>>>>>>> can
>> > > >> > > >>>>>>>>>>>> take a
>> > > >> > > >>>>>>>>>>>>>>> look
>> > > >> > > >>>>>>>>>>>>>>>>> into these issues.
>> > > >> > > >>>>>>>>>>>>>>>>>
>> > > >> > > >>>>>>>>>>>>>>>>> -
>> > > >> > > >>> https://issues.apache.org/jira/browse/FLINK-17159
>> > > >> > > >>>>>> (ES6)
>> > > >> > > >>>>>>>>>>>>>>>>> -
>> > > >> > > >>> https://issues.apache.org/jira/browse/FLINK-17912
>> > > >> > > >>>>>>> (Kafka)
>> > > >> > > >>>>>>>>>>>>>>>>> -
>> > > >> > > >>> https://issues.apache.org/jira/browse/FLINK-17949
>> > > >> > > >>>>>>> (Kafka)
>> > > >> > > >>>>>>>>>>>>>>>>> ⁃
>> > > >> > > >>> https://issues.apache.org/jira/browse/FLINK-18444
>> > > >> > > >>>>>>> (Kafka)
>> > > >> > > >>>>>>>>>>>>>>>>> -
>> > > >> > > >>> https://issues.apache.org/jira/browse/FLINK-18634
>> > > >> > > >>>>>>> (Kafka)
>> > > >> > > >>>>>>>>>>>>>>>>> -
>> > > >> > > >>> https://issues.apache.org/jira/browse/FLINK-18648
>> > > >> > > >>>>>>> (Kafka)
>> > > >> > > >>>>>>>>>>>>>>>>> -
>> > > >> > > >>> https://issues.apache.org/jira/browse/FLINK-18807
>> > > >> > > >>>>>>> (Kafka)
>> > > >> > > >>>>>>>>>>>>>>>>> -
>> > > >> > > >>> https://issues.apache.org/jira/browse/FLINK-19369
>> > > >> > > >>>>>>>>>>>> (BlobClientTest)
>> > > >> > > >>>>>>>>>>>>>>>>> -
>> > > >> > > >>> https://issues.apache.org/jira/browse/FLINK-19436
>> > > >> > > >>>>>>> (TPCDS)
>> > > >> > > >>>>>>>>>>>>>>>>> -
>> > > >> > > >>> https://issues.apache.org/jira/browse/FLINK-19690
>> > > >> > > >>>>>>>>>>>> (Format/Parquet)
>> > > >> > > >>>>>>>>>>>>>>>>> -
>> > > >> > > >>> https://issues.apache.org/jira/browse/FLINK-19775
>> > > >> > > >>>>>>>>>>>>>>>>> (SystemProcessingTimeServiceTest)
>> > > >> > > >>>>>>>>>>>>>>>>>
>> > > >> > > >>>>>>>>>>>>>>>>> Thank you~
>> > > >> > > >>>>>>>>>>>>>>>>>
>> > > >> > > >>>>>>>>>>>>>>>>> Xintong Song
>> > > >> > > >>>>>>>>>>>>>>>>>
>> > > >> > > >>>>>>>>>>>>>>>>>
>> > > >> > > >>>>>>>>>>>>>>>>>
>> > > >> > > >>>>>>>>>>>>>>>>> On Thu, Oct 29, 2020 at 10:21 AM Jingsong Li
>> <
>> > > >> > > >>>>>>>>>>>> jingsonglee0@gmail.com
>> > > >> > > >>>>>>>>>>>>>>
>> > > >> > > >>>>>>>>>>>>>>>>> wrote:
>> > > >> > > >>>>>>>>>>>>>>>>>
>> > > >> > > >>>>>>>>>>>>>>>>>> +1 to backport the FLIP-27 adjustments to
>> > > >> > > >>> 1.11.x.
>> > > >> > > >>>>>>>>>>>>>>>>>>
>> > > >> > > >>>>>>>>>>>>>>>>>> If possible, that would be great. Many
>> people
>> > > >> > > >>> are
>> > > >> > > >>>>>>> looking
>> > > >> > > >>>>>>>>>>> forward
>> > > >> > > >>>>>>>>>>>>> to
>> > > >> > > >>>>>>>>>>>>>>> the
>> > > >> > > >>>>>>>>>>>>>>>>>> FLIP-27 interface, but they don't want to
>> take
>> > > >> > > >>> the
>> > > >> > > >>>>>> risk
>> > > >> > > >>>>>>> to
>> > > >> > > >>>>>>>>>>>> upgrade
>> > > >> > > >>>>>>>>>>>>> to
>> > > >> > > >>>>>>>>>>>>>>>>> 1.12
>> > > >> > > >>>>>>>>>>>>>>>>>> (And wait 1.12). After all, 1.11 is a
>> > > >> > > >> relatively
>> > > >> > > >>>>>> stable
>> > > >> > > >>>>>>>>>>> version.
>> > > >> > > >>>>>>>>>>>>>>>>>>
>> > > >> > > >>>>>>>>>>>>>>>>>> Best,
>> > > >> > > >>>>>>>>>>>>>>>>>> Jingsong
>> > > >> > > >>>>>>>>>>>>>>>>>>
>> > > >> > > >>>>>>>>>>>>>>>>>> On Thu, Oct 29, 2020 at 1:24 AM Stephan
>> Ewen <
>> > > >> > > >>>>>>>>>> sewen@apache.org
>> > > >> > > >>>>>>>>>>>>
>> > > >> > > >>>>>>>>>>>>>>> wrote:
>> > > >> > > >>>>>>>>>>>>>>>>>>
>> > > >> > > >>>>>>>>>>>>>>>>>>> Thanks for starting this.
>> > > >> > > >>>>>>>>>>>>>>>>>>>
>> > > >> > > >>>>>>>>>>>>>>>>>>> +1 form my side to backport the FLIP-27
>> > > >> > > >>>>> adjustments
>> > > >> > > >>>>>> to
>> > > >> > > >>>>>>>>>>> 1.11.x.
>> > > >> > > >>>>>>>>>>>>>>>>>>>
>> > > >> > > >>>>>>>>>>>>>>>>>>> There were quite a few changes, and I think
>> > > >> > > >> we
>> > > >> > > >>>>> need
>> > > >> > > >>>>>> to
>> > > >> > > >>>>>>>>>>>>> cherry-pick
>> > > >> > > >>>>>>>>>>>>>>>>> them
>> > > >> > > >>>>>>>>>>>>>>>>>> all
>> > > >> > > >>>>>>>>>>>>>>>>>>> to not get some inconsistent mix of changes
>> > > >> > > >>> and
>> > > >> > > >>>>> many
>> > > >> > > >>>>>>>>> merge
>> > > >> > > >>>>>>>>>>>>>>> conflicts.
>> > > >> > > >>>>>>>>>>>>>>>>>>> I made a list below of what we need to add
>> > > >> > > >> to
>> > > >> > > >>>>>>>>>> "release-1.11".
>> > > >> > > >>>>>>>>>>>>>>>>>>>
>> > > >> > > >>>>>>>>>>>>>>>>>>> *#### Core Source API Changes to backport
>> > > >> > > >> (in
>> > > >> > > >>>>>> REVERSE
>> > > >> > > >>>>>>>>>> order)*
>> > > >> > > >>>>>>>>>>>>>>>>>>>
>> > > >> > > >>>>>>>>>>>>>>>>>>>  (Use: "git log
>> > > >> > > >>>>>>>>>>>>>>>>>>>
>> > > >> > > >>>>>>>>>>>>
>> > > >> > > >>>>>
>> > > flink-core/src/main/java/org/apache/flink/api/connector/source")
>> > > >> > > >>>>>>>>>>>>>>>>>>>
>> > > >> > > >>>>>>>>>>>>>>>>>>> commit
>> > > >> > > >>> 162c072e9265a7b6dd9d6f5459eb7974091c4c4e
>> > > >> > > >>>>>>>>>>>>>>>>>>> [FLINK-19492][core] Consolidate Source
>> > > >> > > >> Events
>> > > >> > > >>>>>> between
>> > > >> > > >>>>>>>>>> Source
>> > > >> > > >>>>>>>>>>>> API
>> > > >> > > >>>>>>>>>>>>>>> and
>> > > >> > > >>>>>>>>>>>>>>>>>> Split
>> > > >> > > >>>>>>>>>>>>>>>>>>> Reader API
>> > > >> > > >>>>>>>>>>>>>>>>>>>
>> > > >> > > >>>>>>>>>>>>>>>>>>> commit
>> > > >> > > >>> 428fbbad85379f71a9fbf1a1ab3e3dcfa9006436
>> > > >> > > >>>>>>>>>>>>>>>>>>> [FLINK-19205][core] Add access to
>> > > >> > > >>> configuration
>> > > >> > > >>>>> and
>> > > >> > > >>>>>>>>>> hostname
>> > > >> > > >>>>>>>>>>> in
>> > > >> > > >>>>>>>>>>>>> the
>> > > >> > > >>>>>>>>>>>>>>>>>>> SourceReaderContext
>> > > >> > > >>>>>>>>>>>>>>>>>>>
>> > > >> > > >>>>>>>>>>>>>>>>>>> commit
>> > > >> > > >>> 723e1790744ddc78d19c7c978442af1383f38d33
>> > > >> > > >>>>>>>>>>>>>>>>>>> [hotfix][core] Add to Source Enumerator
>> > > >> > > >>>>> convenience
>> > > >> > > >>>>>>>>> methods
>> > > >> > > >>>>>>>>>>> to
>> > > >> > > >>>>>>>>>>>>>>> assign
>> > > >> > > >>>>>>>>>>>>>>>>>>> single split
>> > > >> > > >>>>>>>>>>>>>>>>>>>
>> > > >> > > >>>>>>>>>>>>>>>>>>> *#### Connector Base Changes to Backport
>> (in
>> > > >> > > >>>>> REVERSE
>> > > >> > > >>>>>>>>>> order)*
>> > > >> > > >>>>>>>>>>>>>>>>>>>
>> > > >> > > >>>>>>>>>>>>>>>>>>>  (Use: "git log
>> > > >> > > >>>>>>> flink-connectors/flink-connector-base")
>> > > >> > > >>>>>>>>>>>>>>>>>>>
>> > > >> > > >>>>>>>>>>>>>>>>>>> commit
>> > > >> > > >>> 401f56fe9d6b0271260edf9787cdcbfe4d03874d
>> > > >> > > >>>>>>>>>>>>>>>>>>> [FLINK-19427][FLINK-19489][tests] Fix test
>> > > >> > > >>>>>> conditions
>> > > >> > > >>>>>>>>> for
>> > > >> > > >>>>>>>>>>>>>>>>>>>
>> > > >> > > >>>>>>>
>> 'SplitFetcherTest.testNotifiesWhenGoingIdleConcurrent()'
>> > > >> > > >>>>>>>>>>>>>>>>>>>
>> > > >> > > >>>>>>>>>>>>>>>>>>> commit
>> > > >> > > >>> 68c5c2ff779d82a1ff81ffaf60d8a1b283797db1
>> > > >> > > >>>>>>>>>>>>>>>>>>> [FLINK-19448][connector base] Explicitly
>> > > >> > > >> check
>> > > >> > > >>>> for
>> > > >> > > >>>>>>>>>>> un-expected
>> > > >> > > >>>>>>>>>>>>>>>>> condition
>> > > >> > > >>>>>>>>>>>>>>>>>>> that would leave an inconsistent state
>> > > >> > > >>>>>>>>>>>>>>>>>>>
>> > > >> > > >>>>>>>>>>>>>>>>>>> commit
>> > > >> > > >>> 162c072e9265a7b6dd9d6f5459eb7974091c4c4e
>> > > >> > > >>>>>>>>>>>>>>>>>>> [FLINK-19492][core] Consolidate Source
>> > > >> > > >> Events
>> > > >> > > >>>>>> between
>> > > >> > > >>>>>>>>>> Source
>> > > >> > > >>>>>>>>>>>> API
>> > > >> > > >>>>>>>>>>>>>>> and
>> > > >> > > >>>>>>>>>>>>>>>>>> Split
>> > > >> > > >>>>>>>>>>>>>>>>>>> Reader API
>> > > >> > > >>>>>>>>>>>>>>>>>>>
>> > > >> > > >>>>>>>>>>>>>>>>>>> commit
>> > > >> > > >>> c1ca7a4c7c21ec8868c14cdddd43c559625b794c
>> > > >> > > >>>>>>>>>>>>>>>>>>> [refactor][tests] Move some source test
>> > > >> > > >> utils
>> > > >> > > >>>> from
>> > > >> > > >>>>>>>>>>>>>>>>> flink-connector-base
>> > > >> > > >>>>>>>>>>>>>>>>>> to
>> > > >> > > >>>>>>>>>>>>>>>>>>> flink-core
>> > > >> > > >>>>>>>>>>>>>>>>>>>
>> > > >> > > >>>>>>>>>>>>>>>>>>> commit
>> > > >> > > >>> ee5c4c211c35c70d28252363bbc8400453609977
>> > > >> > > >>>>>>>>>>>>>>>>>>> [FLINK-19251][connectors] Avoid confusing
>> > > >> > > >>> queue
>> > > >> > > >>>>>>>>> handling in
>> > > >> > > >>>>>>>>>>>>>>>>>>> "SplitReader.handleSplitsChanges()"
>> > > >> > > >>>>>>>>>>>>>>>>>>>
>> > > >> > > >>>>>>>>>>>>>>>>>>> commit
>> > > >> > > >>> 5abef56b2bf85bcac786f6b16b6899b6cced7176
>> > > >> > > >>>>>>>>>>>>>>>>>>> [FLINK-19250][connectors] Fix error
>> > > >> > > >>> propagation
>> > > >> > > >>>> in
>> > > >> > > >>>>>>>>>> connector
>> > > >> > > >>>>>>>>>>>> base
>> > > >> > > >>>>>>>>>>>>>>>>>>> (SplitFetcherManager).
>> > > >> > > >>>>>>>>>>>>>>>>>>>
>> > > >> > > >>>>>>>>>>>>>>>>>>> commit
>> > > >> > > >>> 8fcca837c55a9216595ee4c03038b52747098dbb
>> > > >> > > >>>>>>>>>>>>>>>>>>> [hotfix][connectors] Improve JavaDocs for
>> > > >> > > >>>>>>>>>>>>>>> SingleThreadFetcherManager
>> > > >> > > >>>>>>>>>>>>>>>>>>>
>> > > >> > > >>>>>>>>>>>>>>>>>>> commit
>> > > >> > > >>> 4700bb5dde3303cbe98882f6beb7379425717b01
>> > > >> > > >>>>>>>>>>>>>>>>>>> [FLINK-19225][connectors] Various small
>> > > >> > > >>>>> improvements
>> > > >> > > >>>>>>> to
>> > > >> > > >>>>>>>>>>>>>>>>> SourceReaderBase
>> > > >> > > >>>>>>>>>>>>>>>>>>> (part 2)
>> > > >> > > >>>>>>>>>>>>>>>>>>>
>> > > >> > > >>>>>>>>>>>>>>>>>>> commit
>> > > >> > > >>> 12261c6b7ed6478a9b9f6a69cb58246b83cab9b7
>> > > >> > > >>>>>>>>>>>>>>>>>>> [FLINK-17393][connectors] (follow-up)
>> Wakeup
>> > > >> > > >>> the
>> > > >> > > >>>>>>>>>>> SplitFetchers
>> > > >> > > >>>>>>>>>>>>> more
>> > > >> > > >>>>>>>>>>>>>>>>>>> elegantly.
>> > > >> > > >>>>>>>>>>>>>>>>>>>
>> > > >> > > >>>>>>>>>>>>>>>>>>> commit
>> > > >> > > >>> c60aaff0249bfd6b5871b7f82e03efc487a54d6b
>> > > >> > > >>>>>>>>>>>>>>>>>>> [hotfix][tests] Extend test coverage for
>> > > >> > > >>>>>>>>>>>>>>> FutureCompletingBlockingQueue
>> > > >> > > >>>>>>>>>>>>>>>>>>>
>> > > >> > > >>>>>>>>>>>>>>>>>>> commit
>> > > >> > > >>> cef8a587d7fd2fe64cc644da5ed095d82e46f631
>> > > >> > > >>>>>>>>>>>>>>>>>>> [FLINK-19245][connectors] Set default
>> > > >> > > >> capacity
>> > > >> > > >>>> for
>> > > >> > > >>>>>>>>>>>>>>>>>>> FutureCompletingBlockingQueue.
>> > > >> > > >>>>>>>>>>>>>>>>>>>
>> > > >> > > >>>>>>>>>>>>>>>>>>> commit
>> > > >> > > >>> 4ea95782b4c6a2538153d4d16ad3f4839c7de0fb
>> > > >> > > >>>>>>>>>>>>>>>>>>> [FLINK-19223][connectors] Simplify
>> > > >> > > >>> Availability
>> > > >> > > >>>>>> Future
>> > > >> > > >>>>>>>>>> Model
>> > > >> > > >>>>>>>>>>> in
>> > > >> > > >>>>>>>>>>>>>>> Base
>> > > >> > > >>>>>>>>>>>>>>>>>>> Connector
>> > > >> > > >>>>>>>>>>>>>>>>>>>
>> > > >> > > >>>>>>>>>>>>>>>>>>> commit
>> > > >> > > >>> 511857049ba30c8ff0ee56da551fa4a479dc583e
>> > > >> > > >>>>>>>>>>>>>>>>>>> [FLINK-18128][connectors] Ensure idle split
>> > > >> > > >>>>> fetchers
>> > > >> > > >>>>>>>>> lead
>> > > >> > > >>>>>>>>>> to
>> > > >> > > >>>>>>>>>>>>>>>>> availability
>> > > >> > > >>>>>>>>>>>>>>>>>>> notifications.
>> > > >> > > >>>>>>>>>>>>>>>>>>>
>> > > >> > > >>>>>>>>>>>>>>>>>>> commit
>> > > >> > > >>> a8206467af0830dcb89623ea068b5ca3b3450c92
>> > > >> > > >>>>>>>>>>>>>>>>>>> [refactor][core] Eagerly initialize the
>> > > >> > > >>>> FetchTask
>> > > >> > > >>>>> to
>> > > >> > > >>>>>>>>>> support
>> > > >> > > >>>>>>>>>>>>> proper
>> > > >> > > >>>>>>>>>>>>>>>>> unit
>> > > >> > > >>>>>>>>>>>>>>>>>>> testing
>> > > >> > > >>>>>>>>>>>>>>>>>>>
>> > > >> > > >>>>>>>>>>>>>>>>>>> commit
>> > > >> > > >>> 3b2f54bcb437f98e6137c904045cc51072b5c06b
>> > > >> > > >>>>>>>>>>>>>>>>>>> [hotfix][tests] Move constants in
>> > > >> > > >>>> SplitFetcherTest
>> > > >> > > >>>>>>>>> relevant
>> > > >> > > >>>>>>>>>>> to
>> > > >> > > >>>>>>>>>>>>> only
>> > > >> > > >>>>>>>>>>>>>>>>> one
>> > > >> > > >>>>>>>>>>>>>>>>>>> test into test method
>> > > >> > > >>>>>>>>>>>>>>>>>>>
>> > > >> > > >>>>>>>>>>>>>>>>>>> commit
>> > > >> > > >>> d7625760a75a508bf05bcddc380bb4d62ee1743e
>> > > >> > > >>>>>>>>>>>>>>>>>>> [FLINK-19225][connectors] Various small
>> > > >> > > >>>>> improvements
>> > > >> > > >>>>>>> to
>> > > >> > > >>>>>>>>>>>>>>>>> SourceReaderBase
>> > > >> > > >>>>>>>>>>>>>>>>>>>
>> > > >> > > >>>>>>>>>>>>>>>>>>> commit
>> > > >> > > >>> a5b0d3297748c1be47ad579a88f24df2255a8df1
>> > > >> > > >>>>>>>>>>>>>>>>>>> [FLINK-17393][connectors] Wakeup the
>> > > >> > > >>>> SplitFetchers
>> > > >> > > >>>>>>> more
>> > > >> > > >>>>>>>>>>>>> elegantly.
>> > > >> > > >>>>>>>>>>>>>>>>>>>
>> > > >> > > >>>>>>>>>>>>>>>>>>> commit
>> > > >> > > >>> f42a3ebc3e81a034b7221a803c153636fef34903
>> > > >> > > >>>>>>>>>>>>>>>>>>> [FLINK-18680][connectors] Make connector
>> > > >> > > >> base
>> > > >> > > >>>>>>>>>>>> RecordsWithSplitIds
>> > > >> > > >>>>>>>>>>>>>>> more
>> > > >> > > >>>>>>>>>>>>>>>>>>> lightweight.
>> > > >> > > >>>>>>>>>>>>>>>>>>>
>> > > >> > > >>>>>>>>>>>>>>>>>>> commit
>> > > >> > > >>> e3d273de822b085183d09b275a445879ff94b350
>> > > >> > > >>>>>>>>>>>>>>>>>>> [FLINK-19162][connectors] Add 'recycle()'
>> to
>> > > >> > > >>> the
>> > > >> > > >>>>>>>>>>>>>>> RecordsWithSplitIds
>> > > >> > > >>>>>>>>>>>>>>>>> to
>> > > >> > > >>>>>>>>>>>>>>>>>>> support reuse of heavy objects.
>> > > >> > > >>>>>>>>>>>>>>>>>>>
>> > > >> > > >>>>>>>>>>>>>>>>>>> commit
>> > > >> > > >>> 8ebc464c2520453a70001cd712abc8dee6ee89e0
>> > > >> > > >>>>>>>>>>>>>>>>>>> [hotfix][testing] Add a set of
>> > > >> > > >> parameterizable
>> > > >> > > >>>>>> testing
>> > > >> > > >>>>>>>>>> mocks
>> > > >> > > >>>>>>>>>>>> for
>> > > >> > > >>>>>>>>>>>>>>> the
>> > > >> > > >>>>>>>>>>>>>>>>>> Split
>> > > >> > > >>>>>>>>>>>>>>>>>>> Reader API
>> > > >> > > >>>>>>>>>>>>>>>>>>>
>> > > >> > > >>>>>>>>>>>>>>>>>>> commit
>> > > >> > > >>> 930a07438be1185388d7150640f294dfe2a5d378
>> > > >> > > >>>>>>>>>>>>>>>>>>> [hotfix][connectors] Add RequestSplitEvent
>> > > >> > > >> to
>> > > >> > > >>>>>>>>>>>>>>> 'flink-connector-base'
>> > > >> > > >>>>>>>>>>>>>>>>>>>
>> > > >> > > >>>>>>>>>>>>>>>>>>> commit
>> > > >> > > >>> e945ce8a933bc378844782f784ca473c767ca159
>> > > >> > > >>>>>>>>>>>>>>>>>>> [refactor][connectors] Backport of the
>> > > >> > > >>>>>> connector-base
>> > > >> > > >>>>>>>>>>> exception
>> > > >> > > >>>>>>>>>>>>>>>>> handling
>> > > >> > > >>>>>>>>>>>>>>>>>>> from the Kafka Connector Pull Request
>> > > >> > > >>>>>>>>>>>>>>>>>>>
>> > > >> > > >>>>>>>>>>>>>>>>>>>
>> > > >> > > >>>>>>>>>>>>>>>>>>> On Wed, Oct 28, 2020 at 5:49 PM Steven Wu <
>> > > >> > > >>>>>>>>>>>> stevenz3wu@gmail.com>
>> > > >> > > >>>>>>>>>>>>>>>>> wrote:
>> > > >> > > >>>>>>>>>>>>>>>>>>>
>> > > >> > > >>>>>>>>>>>>>>>>>>>> I would love to see this FLIP-27 source
>> > > >> > > >>>>> interface
>> > > >> > > >>>>>>>>>>> improvement
>> > > >> > > >>>>>>>>>>>>> [1]
>> > > >> > > >>>>>>>>>>>>>>>>> made
>> > > >> > > >>>>>>>>>>>>>>>>>> to
>> > > >> > > >>>>>>>>>>>>>>>>>>>> 1.11.3.
>> > > >> > > >>>>>>>>>>>>>>>>>>>>
>> > > >> > > >>>>>>>>>>>>>>>>>>>> [1]
>> > > >> > > >>>>>>> https://issues.apache.org/jira/browse/FLINK-19698
>> > > >> > > >>>>>>>>>>>>>>>>>>>>
>> > > >> > > >>>>>>>>>>>>>>>>>>>> On Wed, Oct 28, 2020 at 12:32 AM Tzu-Li
>> > > >> > > >>>> (Gordon)
>> > > >> > > >>>>>>> Tai <
>> > > >> > > >>>>>>>>>>>>>>>>>>> tzulitai@apache.org>
>> > > >> > > >>>>>>>>>>>>>>>>>>>> wrote:
>> > > >> > > >>>>>>>>>>>>>>>>>>>>
>> > > >> > > >>>>>>>>>>>>>>>>>>>>> Thanks for the replies so far!
>> > > >> > > >>>>>>>>>>>>>>>>>>>>>
>> > > >> > > >>>>>>>>>>>>>>>>>>>>> Just to provide a brief update on the
>> > > >> > > >>> status
>> > > >> > > >>>>> of
>> > > >> > > >>>>>>>>>> blockers
>> > > >> > > >>>>>>>>>>>> for
>> > > >> > > >>>>>>>>>>>>>>>>> 1.11.3
>> > > >> > > >>>>>>>>>>>>>>>>>> so
>> > > >> > > >>>>>>>>>>>>>>>>>>>> far:
>> > > >> > > >>>>>>>>>>>>>>>>>>>>>
>> > > >> > > >>>>>>>>>>>>>>>>>>>>>
>> > > >> > > >>>>>>>>>>>>>>>>>>>>> *PR opened, pending reviewer*-
>> > > >> > > >>> [FLINK-19717]
>> > > >> > > >>>>>>>>>>>>>>>>>> SourceReaderBase.pollNext
>> > > >> > > >>>>>>>>>>>>>>>>>>>> may
>> > > >> > > >>>>>>>>>>>>>>>>>>>>> return END_OF_INPUT if SplitReader.fetch
>> > > >> > > >>>>> throws
>> > > >> > > >>>>>> (
>> > > >> > > >>>>>>>>>>>>>>>>>>>>>
>> > > >> > > >>> https://github.com/apache/flink/pull/13776)
>> > > >> > > >>>>>>>>>>>>>>>>>>>>>
>> > > >> > > >>>>>>>>>>>>>>>>>>>>> *PR opened, reviewed + close to being
>> > > >> > > >>>> merged*
>> > > >> > > >>>>>>>>>>>>>>>>>>>>> - [FLINK-19741] Timer service should
>> > > >> > > >> skip
>> > > >> > > >>>>>>> restoring
>> > > >> > > >>>>>>>>>> from
>> > > >> > > >>>>>>>>>>>> raw
>> > > >> > > >>>>>>>>>>>>>>> keyed
>> > > >> > > >>>>>>>>>>>>>>>>>>> stream
>> > > >> > > >>>>>>>>>>>>>>>>>>>>> if it isn't the writer (
>> > > >> > > >>>>>>>>>>>>>>>>> https://github.com/apache/flink/pull/13761)
>> > > >> > > >>>>>>>>>>>>>>>>>>>>> - [FLINK-19748] Raw keyed stream key
>> > > >> > > >> group
>> > > >> > > >>>>>>> iterator
>> > > >> > > >>>>>>>>>>> should
>> > > >> > > >>>>>>>>>>>> be
>> > > >> > > >>>>>>>>>>>>>>>>>> skipping
>> > > >> > > >>>>>>>>>>>>>>>>>>>>> unwritten key groups (
>> > > >> > > >>>>>>>>>>>>>>> https://github.com/apache/flink/pull/13772)
>> > > >> > > >>>>>>>>>>>>>>>>>>>>>
>> > > >> > > >>>>>>>>>>>>>>>>>>>>> *Merged*
>> > > >> > > >>>>>>>>>>>>>>>>>>>>> - [FLINK-19154] Application mode deletes
>> > > >> > > >>> HA
>> > > >> > > >>>>> data
>> > > >> > > >>>>>>> in
>> > > >> > > >>>>>>>>>> case
>> > > >> > > >>>>>>>>>>> of
>> > > >> > > >>>>>>>>>>>>>>>>> suspended
>> > > >> > > >>>>>>>>>>>>>>>>>>>>> ZooKeeper connection
>> > > >> > > >>>>>>>>>>>>>>>>>>>>> - [FLINK-19569] Upgrade ICU4J to 67.1+
>> > > >> > > >>>>>>>>>>>>>>>>>>>>>
>> > > >> > > >>>>>>>>>>>>>>>>>>>>> Right now as it seems, progress is
>> > > >> > > >> mainly
>> > > >> > > >>>>>> blocked
>> > > >> > > >>>>>>>>> on a
>> > > >> > > >>>>>>>>>>>>> reviewer
>> > > >> > > >>>>>>>>>>>>>>>>> for
>> > > >> > > >>>>>>>>>>>>>>>>>>>>> FLINK-19717.
>> > > >> > > >>>>>>>>>>>>>>>>>>>>> Meanwhile, Xintong is keeping an eye on
>> > > >> > > >>> test
>> > > >> > > >>>>>>>>>>> instabilities
>> > > >> > > >>>>>>>>>>>>> [1]
>> > > >> > > >>>>>>>>>>>>>>> to
>> > > >> > > >>>>>>>>>>>>>>>>> see
>> > > >> > > >>>>>>>>>>>>>>>>>>> if
>> > > >> > > >>>>>>>>>>>>>>>>>>>>> there are any fixes that should be
>> > > >> > > >> applied
>> > > >> > > >>>> to
>> > > >> > > >>>>>>>>>>>> `release-1.11`.
>> > > >> > > >>>>>>>>>>>>>>>>>>>>>
>> > > >> > > >>>>>>>>>>>>>>>>>>>>> This is also a reminder, that if there
>> > > >> > > >> are
>> > > >> > > >>>>> other
>> > > >> > > >>>>>>>>>> blockers
>> > > >> > > >>>>>>>>>>>>> that
>> > > >> > > >>>>>>>>>>>>>>> we
>> > > >> > > >>>>>>>>>>>>>>>>>> need
>> > > >> > > >>>>>>>>>>>>>>>>>>> to
>> > > >> > > >>>>>>>>>>>>>>>>>>>>> be aware of, or a need to re-establish
>> > > >> > > >>>>> estimated
>> > > >> > > >>>>>>>>> time
>> > > >> > > >>>>>>>>>> for
>> > > >> > > >>>>>>>>>>>>>>> getting
>> > > >> > > >>>>>>>>>>>>>>>>>> fixes
>> > > >> > > >>>>>>>>>>>>>>>>>>>> in
>> > > >> > > >>>>>>>>>>>>>>>>>>>>> and delay the RC for 1.11.3, please do
>> > > >> > > >> let
>> > > >> > > >>>> us
>> > > >> > > >>>>>>> know!
>> > > >> > > >>>>>>>>>>>>>>>>>>>>>
>> > > >> > > >>>>>>>>>>>>>>>>>>>>> Cheers,
>> > > >> > > >>>>>>>>>>>>>>>>>>>>> Gordon
>> > > >> > > >>>>>>>>>>>>>>>>>>>>>
>> > > >> > > >>>>>>>>>>>>>>>>>>>>> [1]
>> > > >> > > >>>>>>>>>>>>>>>>>>>>>
>> > > >> > > >>>>>>>>>>>>>>>>>>>>>
>> > > >> > > >>>>>>>>>>>>>>>>>>>>
>> > > >> > > >>>>>>>>>>>>>>>>>>>
>> > > >> > > >>>>>>>>>>>>>>>>>>
>> > > >> > > >>>>>>>>>>>>>>>>>
>> > > >> > > >>>>>>>>>>>>>>>
>> > > >> > > >>>>>>>>>>>>>
>> > > >> > > >>>>>>>>>>>>
>> > > >> > > >>>>>>>>>>>
>> > > >> > > >>>>>>>>>>
>> > > >> > > >>>>>>>>>
>> > > >> > > >>>>>>>
>> > > >> > > >>>>>>
>> > > >> > > >>>>>
>> > > >> > > >>>>
>> > > >> > > >>>
>> > > >> > > >>
>> > > >> > >
>> > > >> >
>> > > >>
>> > >
>> >
>> https://issues.apache.org/jira/issues/?jql=project%20%3D%20FLINK%20AND%20status%20in%20(Open%2C%20%22In%20Progress%22%2C%20Reopened)%20AND%20affectedVersion%20in%20(1.11.0%2C%201.11.1%2C%201.11.2)%20AND%20labels%20%3D%20test-stability
>> > > >> > > >>>>>>>>>>>>>>>>>>>>>
>> > > >> > > >>>>>>>>>>>>>>>>>>>>> On Mon, Oct 26, 2020 at 9:43 PM Kostas
>> > > >> > > >>>>> Kloudas <
>> > > >> > > >>>>>>>>>>>>>>>>> kkloudas@apache.org>
>> > > >> > > >>>>>>>>>>>>>>>>>>>>> wrote:
>> > > >> > > >>>>>>>>>>>>>>>>>>>>>
>> > > >> > > >>>>>>>>>>>>>>>>>>>>>> +1 for releasing Flink 1.11.3 as it
>> > > >> > > >>>>> contains a
>> > > >> > > >>>>>>>>> number
>> > > >> > > >>>>>>>>>>> of
>> > > >> > > >>>>>>>>>>>>>>>>> important
>> > > >> > > >>>>>>>>>>>>>>>>>>>>>> fixes and thanks Gordon and Xintong
>> > > >> > > >> for
>> > > >> > > >>>>>>>>> volunteering.
>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>
>> > > >> > > >>>>>>>>>>>>>>>>>>>>>> Cheers,
>> > > >> > > >>>>>>>>>>>>>>>>>>>>>> Kostas
>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>
>> > > >> > > >>>>>>>>>>>>>>>>>>>>>> On Mon, Oct 26, 2020 at 4:37 AM Yu Li
>> > > >> > > >> <
>> > > >> > > >>>>>>>>>>> carp84@gmail.com>
>> > > >> > > >>>>>>>>>>>>>>> wrote:
>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>
>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>> +1 for releasing Flink 1.11.3, and
>> > > >> > > >>>> thanks
>> > > >> > > >>>>>>> Gordon
>> > > >> > > >>>>>>>>>> and
>> > > >> > > >>>>>>>>>>>>>>> Xintong
>> > > >> > > >>>>>>>>>>>>>>>>> for
>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>> volunteering as our release
>> > > >> > > >> managers.
>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>
>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>> Best Regards,
>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>> Yu
>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>
>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>
>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>> On Mon, 26 Oct 2020 at 09:45,
>> > > >> > > >> Xintong
>> > > >> > > >>>>> Song <
>> > > >> > > >>>>>>>>>>>>>>>>>> tonysong820@gmail.com>
>> > > >> > > >>>>>>>>>>>>>>>>>>>>>> wrote:
>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>
>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>> Thanks Gordan for starting this
>> > > >> > > >>>>>> discussion.
>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>> My pleasure to help with the
>> > > >> > > >> release
>> > > >> > > >>>>>>> process.
>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>
>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>> Thank you~
>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>
>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>> Xintong Song
>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>
>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>
>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>
>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>> On Fri, Oct 23, 2020 at 11:29 PM
>> > > >> > > >>> Till
>> > > >> > > >>>>>>>>> Rohrmann <
>> > > >> > > >>>>>>>>>>>>>>>>>>>> trohrmann@apache.org
>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>
>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>> wrote:
>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>
>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>> Thanks for starting this
>> > > >> > > >>> discussion
>> > > >> > > >>>>>>> Gordon.
>> > > >> > > >>>>>>>>>> There
>> > > >> > > >>>>>>>>>>>> are
>> > > >> > > >>>>>>>>>>>>>>> over
>> > > >> > > >>>>>>>>>>>>>>>>>> 100
>> > > >> > > >>>>>>>>>>>>>>>>>>>>> issues
>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>> which are fixed for 1.11.3.
>> > > >> > > >> Hence
>> > > >> > > >>> +1
>> > > >> > > >>>>>> for a
>> > > >> > > >>>>>>>>>>> soonish
>> > > >> > > >>>>>>>>>>>>>>> 1.11.3
>> > > >> > > >>>>>>>>>>>>>>>>>>>> release.
>> > > >> > > >>>>>>>>>>>>>>>>>>>>>> Thanks
>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>> for volunteering as our release
>> > > >> > > >>>>> managers
>> > > >> > > >>>>>>>>> Gordon
>> > > >> > > >>>>>>>>>>> and
>> > > >> > > >>>>>>>>>>>>>>>>> Xintong!
>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>> Cheers,
>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>> Till
>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>> On Fri, Oct 23, 2020 at 5:02 PM
>> > > >> > > >>>> Tzu-Li
>> > > >> > > >>>>>>>>> (Gordon)
>> > > >> > > >>>>>>>>>>>> Tai <
>> > > >> > > >>>>>>>>>>>>>>>>>>>>>> tzulitai@apache.org
>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>> Hi,
>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>> Xintong and I would like to
>> > > >> > > >>> start a
>> > > >> > > >>>>>>>>> discussion
>> > > >> > > >>>>>>>>>>> for
>> > > >> > > >>>>>>>>>>>>>>>>> releasing
>> > > >> > > >>>>>>>>>>>>>>>>>>>> Flink
>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>> 1.11.3
>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>> soon.
>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>> It seems like we already have a
>> > > >> > > >>> few
>> > > >> > > >>>>>>>>> pressing
>> > > >> > > >>>>>>>>>>>> issues
>> > > >> > > >>>>>>>>>>>>>>> that
>> > > >> > > >>>>>>>>>>>>>>>>>> needs
>> > > >> > > >>>>>>>>>>>>>>>>>>>> to
>> > > >> > > >>>>>>>>>>>>>>>>>>>>> be
>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>> included in a new hotfix
>> > > >> > > >> release:
>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>>   - Heap-based timers’ restore
>> > > >> > > >>>>>> behaviour
>> > > >> > > >>>>>>>>> is
>> > > >> > > >>>>>>>>>>>>> causing a
>> > > >> > > >>>>>>>>>>>>>>>>>>> critical
>> > > >> > > >>>>>>>>>>>>>>>>>>>>>> recovery
>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>>   issue for StateFun [1] [2]
>> > > >> > > >>> [3].
>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>>   - There are several
>> > > >> > > >> robustness
>> > > >> > > >>>>>> issues
>> > > >> > > >>>>>>>>> for
>> > > >> > > >>>>>>>>>> the
>> > > >> > > >>>>>>>>>>>>>>> FLIP-27
>> > > >> > > >>>>>>>>>>>>>>>>> new
>> > > >> > > >>>>>>>>>>>>>>>>>>>>> source
>> > > >> > > >>>>>>>>>>>>>>>>>>>>>> API,
>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>>   such as [4]. We already have
>> > > >> > > >>>> some
>> > > >> > > >>>>>>> users
>> > > >> > > >>>>>>>>>> using
>> > > >> > > >>>>>>>>>>>> the
>> > > >> > > >>>>>>>>>>>>>>>>> FLIP-27
>> > > >> > > >>>>>>>>>>>>>>>>>>> API
>> > > >> > > >>>>>>>>>>>>>>>>>>>>>> with
>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>> 1.11.x,
>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>>   so it would be important to
>> > > >> > > >>> get
>> > > >> > > >>>>>> those
>> > > >> > > >>>>>>>>> fixes
>> > > >> > > >>>>>>>>>>> in
>> > > >> > > >>>>>>>>>>>>> for
>> > > >> > > >>>>>>>>>>>>>>>>> 1.11.x
>> > > >> > > >>>>>>>>>>>>>>>>>>> as
>> > > >> > > >>>>>>>>>>>>>>>>>>>>>> well.
>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>> Apart from the issues that are
>> > > >> > > >>>>> already
>> > > >> > > >>>>>>>>> marked
>> > > >> > > >>>>>>>>>> as
>> > > >> > > >>>>>>>>>>>>>>> blocker
>> > > >> > > >>>>>>>>>>>>>>>>> for
>> > > >> > > >>>>>>>>>>>>>>>>>>>>> 1.11.3
>> > > >> > > >>>>>>>>>>>>>>>>>>>>>> in
>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>> our
>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>> JIRA [5], please let us know in
>> > > >> > > >>>> this
>> > > >> > > >>>>>>>>> thread if
>> > > >> > > >>>>>>>>>>>> there
>> > > >> > > >>>>>>>>>>>>>>> is
>> > > >> > > >>>>>>>>>>>>>>>>>>> already
>> > > >> > > >>>>>>>>>>>>>>>>>>>>>> ongoing
>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>> work for other important fixes
>> > > >> > > >>> that
>> > > >> > > >>>>> we
>> > > >> > > >>>>>>>>> should
>> > > >> > > >>>>>>>>>>> try
>> > > >> > > >>>>>>>>>>>> to
>> > > >> > > >>>>>>>>>>>>>>>>>> include.
>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>> Xintong and I would like to
>> > > >> > > >>>> volunteer
>> > > >> > > >>>>>> for
>> > > >> > > >>>>>>>>>>> managing
>> > > >> > > >>>>>>>>>>>>>>> this
>> > > >> > > >>>>>>>>>>>>>>>>>>> release,
>> > > >> > > >>>>>>>>>>>>>>>>>>>>> and
>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>> will
>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>> try to communicate the priority
>> > > >> > > >>> of
>> > > >> > > >>>>>>> pending
>> > > >> > > >>>>>>>>>>>> blockers
>> > > >> > > >>>>>>>>>>>>>>> over
>> > > >> > > >>>>>>>>>>>>>>>>> the
>> > > >> > > >>>>>>>>>>>>>>>>>>>> next
>> > > >> > > >>>>>>>>>>>>>>>>>>>>>> few
>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>> days.
>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>> Since the aforementioned issues
>> > > >> > > >>> are
>> > > >> > > >>>>>> quite
>> > > >> > > >>>>>>>>>>>> critical,
>> > > >> > > >>>>>>>>>>>>>>> we’d
>> > > >> > > >>>>>>>>>>>>>>>>>> like
>> > > >> > > >>>>>>>>>>>>>>>>>>> to
>> > > >> > > >>>>>>>>>>>>>>>>>>>>> aim
>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>> for a *feature
>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>> freeze by the end of next week
>> > > >> > > >>>> (Oct.
>> > > >> > > >>>>>>> 30th)*
>> > > >> > > >>>>>>>>>> and
>> > > >> > > >>>>>>>>>>>>> start
>> > > >> > > >>>>>>>>>>>>>>> the
>> > > >> > > >>>>>>>>>>>>>>>>>>>> release
>> > > >> > > >>>>>>>>>>>>>>>>>>>>>> voting
>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>> process the week after.
>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>> If that is too short of a
>> > > >> > > >> notice
>> > > >> > > >>>> and
>> > > >> > > >>>>>> you
>> > > >> > > >>>>>>>>> might
>> > > >> > > >>>>>>>>>>>> need
>> > > >> > > >>>>>>>>>>>>>>> more
>> > > >> > > >>>>>>>>>>>>>>>>>> time,
>> > > >> > > >>>>>>>>>>>>>>>>>>>>>> please
>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>> let
>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>> us know!
>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>> Cheers,
>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>> Gordon
>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>> [1]
>> > > >> > > >>>>>>>>>>>>>
>> https://issues.apache.org/jira/browse/FLINK-19692
>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>> [2]
>> > > >> > > >>>>>>>>>>>>>
>> https://issues.apache.org/jira/browse/FLINK-19741
>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>> [3]
>> > > >> > > >>>>>>>>>>>>>
>> https://issues.apache.org/jira/browse/FLINK-19748
>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>> [4]
>> > > >> > > >>>>>>>>>>>>>
>> https://issues.apache.org/jira/browse/FLINK-19717
>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>> [5]
>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>
>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>
>> > > >> > > >>>>>>>>>>>>>>>>>>>>>
>> > > >> > > >>>>>>>>>>>>>>>>>>>>
>> > > >> > > >>>>>>>>>>>>>>>>>>>
>> > > >> > > >>>>>>>>>>>>>>>>>>
>> > > >> > > >>>>>>>>>>>>>>>>>
>> > > >> > > >>>>>>>>>>>>>>>
>> > > >> > > >>>>>>>>>>>>>
>> > > >> > > >>>>>>>>>>>>
>> > > >> > > >>>>>>>>>>>
>> > > >> > > >>>>>>>>>>
>> > > >> > > >>>>>>>>>
>> > > >> > > >>>>>>>
>> > > >> > > >>>>>>
>> > > >> > > >>>>>
>> > > >> > > >>>>
>> > > >> > > >>>
>> > > >> > > >>
>> > > >> > >
>> > > >> >
>> > > >>
>> > >
>> >
>> https://issues.apache.org/jira/issues/?jql=project%20%3D%20FLINK%20AND%20priority%20%3D%20Blocker%20AND%20fixVersion%20%3D%201.11.3
>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>
>> > > >> > > >>>>>>>>>>>>>>>>>>>>>>
>> > > >> > > >>>>>>>>>>>>>>>>>>>>>
>> > > >> > > >>>>>>>>>>>>>>>>>>>>
>> > > >> > > >>>>>>>>>>>>>>>>>>>
>> > > >> > > >>>>>>>>>>>>>>>>>>
>> > > >> > > >>>>>>>>>>>>>>>>>>
>> > > >> > > >>>>>>>>>>>>>>>>>> --
>> > > >> > > >>>>>>>>>>>>>>>>>> Best, Jingsong Lee
>> > > >> > > >>>>>>>>>>>>>>>>>>
>> > > >> > > >>>>>>>>>>>>>>>>>
>> > > >> > > >>>>>>>>>>>>>>>>
>> > > >> > > >>>>>>>>>>>>>>>
>> > > >> > > >>>>>>>>>>>>>>
>> > > >> > > >>>>>>>>>>>>>
>> > > >> > > >>>>>>>>>>>>
>> > > >> > > >>>>>>>>>>>
>> > > >> > > >>>>>>>>>>
>> > > >> > > >>>>>>>>>
>> > > >> > > >>>>>>>>
>> > > >> > > >>>>>>>
>> > > >> > > >>>>>>
>> > > >> > > >>>>>
>> > > >> > > >>>>
>> > > >> > > >>>
>> > > >> > > >>
>> > > >> > >
>> > > >> > >
>> > > >> > >
>> > > >> >
>> > > >>
>> > > >
>> > >
>> >
>>
>

Re: [DISCUSS] Releasing Apache Flink 1.11.3

Posted by Xintong Song <to...@gmail.com>.
Thanks for the notice, Zhu.
Let's have the discussion on the jira ticket.

Thank you~

Xintong Song



On Mon, Nov 23, 2020 at 12:10 PM Zhu Zhu <re...@gmail.com> wrote:

> Hi Xintong,
> We just found an issue FLINK-20285 and I think it is a blocker for 1.11.3.
>
> Thanks,
> Zhu
>
> Gyula Fóra <gy...@gmail.com> 于2020年11月18日周三 下午9:36写道：
>
> > Hi All!
> >
> > I have found the following issue today which might be considered a
> blocker
> > for this release as well:
> >
> > https://issues.apache.org/jira/browse/FLINK-20221
> >
> > Could someone please quickly provide a second set of eyes and confirm
> that
> > this is indeed a big problem? :)
> >
> > Thank you!
> > Gyula
> >
> > On Wed, Nov 18, 2020 at 5:12 AM Xintong Song <to...@gmail.com>
> > wrote:
> >
> > > Hi devs,
> > >
> > > Updates on the progress of preparing the 1.11.3 release.
> > >
> > > We are approaching the creation of our first release candidate. Thanks
> > all
> > > for the efforts so far.
> > >
> > > # Ongoing threads
> > >
> > >    - *FLIP-27 backportings:* 74 out of 102 commits are already
> > backported.
> > >    Stephan and Becket are actively working on this.
> > >    - *Blockers:* There are 2 remaining blockers, both have been fixed
> on
> > >    the master branch and should be backported along with the FLIP-27
> > > changes.
> > >
> > > As soon as the above threads are done, we will create our first release
> > > candidate. If there's any other issues that you believe should be a
> > release
> > > blocker, please reach out to either me or Gordon.
> > >
> > > # JIRA version
> > >
> > > Version 1.11.4 has been created on JIRA. I've already tagged some of
> the
> > > tickets with the new fix version. Please feel free to continue with the
> > > efforts and merge once they are ready. I'll double check and update the
> > fix
> > > versions before the release.
> > >
> > > Thank you~
> > >
> > > Xintong Song
> > >
> > >
> > >
> > > On Thu, Nov 12, 2020 at 1:31 PM Xintong Song <to...@gmail.com>
> > > wrote:
> > >
> > > > Thanks for the notice and fix, Roman.
> > > >
> > > > Thank you~
> > > >
> > > > Xintong Song
> > > >
> > > >
> > > >
> > > > On Wed, Nov 11, 2020 at 5:53 PM Khachatryan Roman <
> > > > khachatryan.roman@gmail.com> wrote:
> > > >
> > > >> Hi,
> > > >>
> > > >> I'd like FLINK-20079 [1] to be merged into 1.11 and included in
> > 1.11.3.
> > > >>
> > > >> [1] https://issues.apache.org/jira/browse/FLINK-20079
> > > >>
> > > >> Regards,
> > > >> Roman
> > > >>
> > > >>
> > > >> On Tue, Nov 10, 2020 at 8:21 AM Xintong Song <tonysong820@gmail.com
> >
> > > >> wrote:
> > > >>
> > > >> > Thanks for the notice, Dian.
> > > >> >
> > > >> > Thank you~
> > > >> >
> > > >> > Xintong Song
> > > >> >
> > > >> >
> > > >> >
> > > >> > On Tue, Nov 10, 2020 at 10:19 AM Dian Fu <di...@gmail.com>
> > > wrote:
> > > >> >
> > > >> > > Hi Xintong,
> > > >> > >
> > > >> > > I want to bring one more issue to your attention [1]. The test
> > case
> > > >> > > UnalignedCheckpointCompatibilityITCase.test failed several times
> > in
> > > >> the
> > > >> > > last nightly test of release-1.11. We need to figure out if it's
> > > just
> > > >> an
> > > >> > > instable test or caused by recent changes.
> > > >> > >
> > > >> > > [1] https://issues.apache.org/jira/browse/FLINK-20065
> > > >> > >
> > > >> > > > 在 2020年11月10日，上午9:24，Xintong Song <to...@gmail.com> 写道：
> > > >> > > >
> > > >> > > > Thanks for the replies.
> > > >> > > >
> > > >> > > > Thank you~
> > > >> > > >
> > > >> > > > Xintong Song
> > > >> > > >
> > > >> > > >
> > > >> > > >
> > > >> > > > On Tue, Nov 10, 2020 at 1:09 AM Becket Qin <
> > becket.qin@gmail.com>
> > > >> > wrote:
> > > >> > > >
> > > >> > > >> Hi Xintong,
> > > >> > > >>
> > > >> > > >> Thanks for driving the release. Just want to sync up on the
> > > FLIP-27
> > > >> > > >> backporting. Stephan and I are still trying to backport a
> bunch
> > > of
> > > >> > > patches
> > > >> > > >> of Source to 1.11.3. Including:
> > > >> > > >>
> > > >> > > >> [FLINK-19698][connector/common] Add a close() method to the
> > > >> > SplitReader.
> > > >> > > >> [FLINK-19717] SourceReaderBase.pollNext may return
> END_OF_INPUT
> > > if
> > > >> > > >> SplitReader.fetch throws
> > > >> > > >> [FLINK-19535] [connector/common] Avoid failing a job multiple
> > > >> times in
> > > >> > > >> SourceCoordinator.
> > > >> > > >> [FLINK-19265] [FLINK-20049][core] Source API final
> adjustments.
> > > >> > > >>
> > > >> > > >> and a few more fixes.
> > > >> > > >>
> > > >> > > >> We are currently trying to fix them in 1.12 first so it might
> > > take
> > > >> a
> > > >> > > little
> > > >> > > >> longer to backport them to 1.11.3. I think it will probably
> > take
> > > >> us a
> > > >> > > few
> > > >> > > >> more days to finish the backport. So that would roughly be
> the
> > > end
> > > >> of
> > > >> > > this
> > > >> > > >> week.
> > > >> > > >>
> > > >> > > >> Thanks,
> > > >> > > >>
> > > >> > > >> Jiangjie (Becket) Qin
> > > >> > > >>
> > > >> > > >>
> > > >> > > >>
> > > >> > > >>
> > > >> > > >> On Mon, Nov 9, 2020 at 9:57 PM Till Rohrmann <
> > > trohrmann@apache.org
> > > >> >
> > > >> > > wrote:
> > > >> > > >>
> > > >> > > >>> Yes, I've downgraded FLINK-19816 to critical.
> > > >> > > >>>
> > > >> > > >>> Cheers,
> > > >> > > >>> Till
> > > >> > > >>>
> > > >> > > >>> On Mon, Nov 9, 2020 at 10:19 AM Xintong Song <
> > > >> tonysong820@gmail.com>
> > > >> > > >>> wrote:
> > > >> > > >>>
> > > >> > > >>>> Thanks for the notice, Till.
> > > >> > > >>>>
> > > >> > > >>>> I just checked and found FLINK-20033 is already fixed.
> Shall
> > we
> > > >> also
> > > >> > > >>>> downgrade FLINK-19816 to `Critical`?
> > > >> > > >>>>
> > > >> > > >>>> Thank you~
> > > >> > > >>>>
> > > >> > > >>>> Xintong Song
> > > >> > > >>>>
> > > >> > > >>>>
> > > >> > > >>>>
> > > >> > > >>>> On Mon, Nov 9, 2020 at 4:42 PM Till Rohrmann <
> > > >> trohrmann@apache.org>
> > > >> > > >>> wrote:
> > > >> > > >>>>
> > > >> > > >>>>> I would like to bring one more critical issue to your
> > > attention
> > > >> > which
> > > >> > > >>> is
> > > >> > > >>>>> FLINK-20033 [1]. I believe that this issue is actually
> > causing
> > > >> what
> > > >> > > >> has
> > > >> > > >>>>> been reported in FLINK-19816 [2]. I hope to have it fixed
> by
> > > the
> > > >> > end
> > > >> > > >> of
> > > >> > > >>>>> today. Once FLINK-20033 is fixed, I think that we don't
> have
> > > to
> > > >> > block
> > > >> > > >>> the
> > > >> > > >>>>> release on FLINK-19816.
> > > >> > > >>>>>
> > > >> > > >>>>> [1] https://issues.apache.org/jira/browse/FLINK-20033
> > > >> > > >>>>> [2] https://issues.apache.org/jira/browse/FLINK-19816
> > > >> > > >>>>>
> > > >> > > >>>>> Cheers,
> > > >> > > >>>>> Till
> > > >> > > >>>>>
> > > >> > > >>>>> On Mon, Nov 9, 2020 at 4:05 AM Xintong Song <
> > > >> tonysong820@gmail.com
> > > >> > >
> > > >> > > >>>> wrote:
> > > >> > > >>>>>
> > > >> > > >>>>>> Hi devs,
> > > >> > > >>>>>>
> > > >> > > >>>>>> I'd like to provide an update on the progress of
> preparing
> > > >> release
> > > >> > > >>>>> 1.11.3.
> > > >> > > >>>>>>
> > > >> > > >>>>>> *Blockers*
> > > >> > > >>>>>> We currently have 3 remaining blockers. (3 resolved and 1
> > > >> emerged
> > > >> > > >>>>> compared
> > > >> > > >>>>>> to last week)
> > > >> > > >>>>>>
> > > >> > > >>>>>> - [FLINK-19698] Add close() method and
> > onCheckpointComplete()
> > > >> to
> > > >> > > >> the
> > > >> > > >>>>>> Source.
> > > >> > > >>>>>> The issue has been fixed on the master branch. It's
> > currently
> > > >> > > >> blocked
> > > >> > > >>>> on
> > > >> > > >>>>>> the FLIP-27 backportings to backport it to the 1.11
> branch.
> > > >> > > >>>>>>
> > > >> > > >>>>>> - [FLINK-19717] SourceReaderBase.pollNext may return
> > > >> END_OF_INPUT
> > > >> > > >> if
> > > >> > > >>>>>> SplitReader.fetch throws
> > > >> > > >>>>>> A PR has been opened and reviewed. From the discussions
> on
> > > the
> > > >> PR,
> > > >> > > >> it
> > > >> > > >>>>> looks
> > > >> > > >>>>>> close to mergeable.
> > > >> > > >>>>>>
> > > >> > > >>>>>> - [FLINK-19816] Flink restored from a wrong checkpoint (a
> > > very
> > > >> old
> > > >> > > >>> one
> > > >> > > >>>>> and
> > > >> > > >>>>>> not the last completed one)
> > > >> > > >>>>>> This is a newly emerged blocker and Matthias is working
> on
> > > it.
> > > >> > > >>>>>>
> > > >> > > >>>>>> *Test Instabilities*
> > > >> > > >>>>>> We currently have 27 test instabilities[1].
> > > >> > > >>>>>> AFAIK, none of them are as serious as to block the 1.11.3
> > > >> release.
> > > >> > > >>>>>>
> > > >> > > >>>>>> *FLIP-27 Backprotings*
> > > >> > > >>>>>>
> > > >> > > >>>>>> I noticed that there's no jira issues opened on the
> FLIP-27
> > > >> > > >>> backporting
> > > >> > > >>>>>> efforts, which is part of the major efforts planned for
> the
> > > >> 1.11.3
> > > >> > > >>>>> release,
> > > >> > > >>>>>> making it hard to track the progress.
> > > >> > > >>>>>>
> > > >> > > >>>>>>
> > > >> > > >>>>>> @Stephan and @Becket, could you please share the updates
> on
> > > the
> > > >> > > >>>>> backporting
> > > >> > > >>>>>> efforts? How is the progress and when are the efforts
> > > expected
> > > >> to
> > > >> > > >> be
> > > >> > > >>>>>> finished? It would be appreciated and helpful if we can
> > have
> > > a
> > > >> > jira
> > > >> > > >>>>> ticket
> > > >> > > >>>>>> to track the backporing progress publicly.
> > > >> > > >>>>>>
> > > >> > > >>>>>>
> > > >> > > >>>>>> Thank you~
> > > >> > > >>>>>>
> > > >> > > >>>>>> Xintong Song
> > > >> > > >>>>>>
> > > >> > > >>>>>>
> > > >> > > >>>>>> [1]
> > > >> > > >>>>>>
> > > >> > > >>>>>>
> > > >> > > >>>>>
> > > >> > > >>>>
> > > >> > > >>>
> > > >> > > >>
> > > >> > >
> > > >> >
> > > >>
> > >
> >
> https://issues.apache.org/jira/browse/FLINK-16908?filter=12348580&jql=project%20%3D%20FLINK%20AND%20resolution%20%3D%20Unresolved%20AND%20affectedVersion%20in%20(1.11.0%2C%201.11.1%2C%201.11.2%2C%201.11.3)%20AND%20labels%20%3D%20test-stability%20ORDER%20BY%20updated%20DESC%2C%20created%20DESC
> > > >> > > >>>>>>
> > > >> > > >>>>>> On Wed, Nov 4, 2020 at 3:58 PM Tzu-Li (Gordon) Tai <
> > > >> > > >>>> tzulitai@apache.org>
> > > >> > > >>>>>> wrote:
> > > >> > > >>>>>>
> > > >> > > >>>>>>> Thanks for the efforts so far with pushing for 1.11.3.
> > > >> > > >>>>>>>
> > > >> > > >>>>>>> I'd like to provide a mid-week update on how we're
> looking
> > > >> with
> > > >> > > >> all
> > > >> > > >>>> the
> > > >> > > >>>>>>> pending blockers and backports:
> > > >> > > >>>>>>>
> > > >> > > >>>>>>> *Blockers**:*
> > > >> > > >>>>>>>
> > > >> > > >>>>>>> Currently all blockers either have an open PR, or have
> > been
> > > >> > > >> merged.
> > > >> > > >>>>> I'll
> > > >> > > >>>>>>> highlight below the blockers *that still require
> reviewing
> > > >> > > >> efforts
> > > >> > > >>>> for
> > > >> > > >>>>>> them
> > > >> > > >>>>>>> to move forward:*
> > > >> > > >>>>>>>
> > > >> > > >>>>>>> - [FLINK-19909] Flink application in attach mode could
> not
> > > >> > > >>> terminate
> > > >> > > >>>>> when
> > > >> > > >>>>>>> the only job is canceled. PR:
> > > >> > > >>>>> https://github.com/apache/flink/pull/13911
> > > >> > > >>>>>>> - [FLINK-19717] SourceReaderBase.pollNext may return
> > > >> END_OF_INPUT
> > > >> > > >>> if
> > > >> > > >>>>>>> SplitReader.fetch throws. PR:
> > > >> > > >>>>> https://github.com/apache/flink/pull/13776
> > > >> > > >>>>>>>
> > > >> > > >>>>>>> The above PRs currently have no reviews at all yet. They
> > > seem
> > > >> to
> > > >> > > >>>>> already
> > > >> > > >>>>>>> have designated reviewers.
> > > >> > > >>>>>>>
> > > >> > > >>>>>>>
> > > >> > > >>>>>>> *Backports:*
> > > >> > > >>>>>>> - FLIP-27 robustness improvement backports: Stephan is
> > > >> currently
> > > >> > > >>>>> working
> > > >> > > >>>>>> on
> > > >> > > >>>>>>> backporting several FLIP-27 changes. There are no PRs
> yet
> > > for
> > > >> the
> > > >> > > >>>>>>> backported changes.
> > > >> > > >>>>>>> - DataStreamUtils.collect() refactorings backport:
> Steven
> > Wu
> > > >> > > >>>> mentioned
> > > >> > > >>>>> to
> > > >> > > >>>>>>> backport this, but AFAIK this isn't assigned to anyone
> > yet.
> > > >> > > >>>>>>>
> > > >> > > >>>>>>> *ETAs*:
> > > >> > > >>>>>>>
> > > >> > > >>>>>>> I'd like to request ETAs for the remaining backports, to
> > > >> prevent
> > > >> > > >> a
> > > >> > > >>>>> creep
> > > >> > > >>>>>> in
> > > >> > > >>>>>>> the scope of this bugfix release.
> > > >> > > >>>>>>>
> > > >> > > >>>>>>> We already have Flink users that would benefit from
> fixes
> > > that
> > > >> > > >> have
> > > >> > > >>>>> been
> > > >> > > >>>>>>> merged to release-1.11 already, so technically speaking
> > the
> > > >> > > >>> backports
> > > >> > > >>>>>>> should be considered "nice-to-have" (to the best of my
> > > >> knowledge
> > > >> > > >> of
> > > >> > > >>>> the
> > > >> > > >>>>>>> changes) and could potentially be moved to a follow-up
> > > 1.11.4.
> > > >> > > >>>>>>> Most notably, the Stateful Functions project is already
> > > >> waiting
> > > >> > > >> on
> > > >> > > >>>>> Flink
> > > >> > > >>>>>>> 1.11.3 to address critical recovery issues (please see
> the
> > > >> > > >> StateFun
> > > >> > > >>>>> 2.2.1
> > > >> > > >>>>>>> release discussion thread [1]).
> > > >> > > >>>>>>>
> > > >> > > >>>>>>> @Stephan Ewen <se...@apache.org> @Becket Qin <
> > > >> > > >> becket.qin@gmail.com
> > > >> > > >>>>
> > > >> > > >>>>>> could
> > > >> > > >>>>>>> you provide an ETA for the FLIP-27 backports? It would
> > help
> > > to
> > > >> > > >> get
> > > >> > > >>> a
> > > >> > > >>>>>> better
> > > >> > > >>>>>>> estimate to decide how we proceed here.
> > > >> > > >>>>>>>
> > > >> > > >>>>>>> Cheers,
> > > >> > > >>>>>>> Gordon
> > > >> > > >>>>>>>
> > > >> > > >>>>>>> [1]
> > > >> > > >>>>>>>
> > > >> > > >>>>>>>
> > > >> > > >>>>>>
> > > >> > > >>>>>
> > > >> > > >>>>
> > > >> > > >>>
> > > >> > > >>
> > > >> > >
> > > >> >
> > > >>
> > >
> >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Releasing-StateFun-hotfix-version-2-2-1-td46239.html
> > > >> > > >>>>>>>
> > > >> > > >>>>>>> On Wed, Nov 4, 2020 at 3:16 PM Tzu-Li (Gordon) Tai <
> > > >> > > >>>>> tzulitai@apache.org>
> > > >> > > >>>>>>> wrote:
> > > >> > > >>>>>>>
> > > >> > > >>>>>>>>> The collect() utils can be picked back, I see no issue
> > > with
> > > >> > > >>> that
> > > >> > > >>>>> (it
> > > >> > > >>>>>> is
> > > >> > > >>>>>>>> isolated utilities).
> > > >> > > >>>>>>>>
> > > >> > > >>>>>>>> Just checking on all the requested backports mentioned
> in
> > > >> this
> > > >> > > >>>>> thread,
> > > >> > > >>>>>>> and
> > > >> > > >>>>>>>> figuring out which ones seem to still be unassigned /
> > open.
> > > >> > > >>>>>>>>
> > > >> > > >>>>>>>> Is someone working on backporting
> > > >> > > >>>>>>>>
> > > >> > > >>>>>>>
> > > >> > > >>>>>>
> > > >> > > >>>>>
> > > >> > > >>>>
> > > >> > > >>>
> > > >> > > >>
> > > >> > >
> > > >> >
> > > >>
> > >
> >
> https://github.com/apache/flink/commit/09a7a66b7313fea64817fe960a8da1265b428efc
> > > >> > > >>>>>>>> to release-1.11 at the moment?
> > > >> > > >>>>>>>>
> > > >> > > >>>>>>>> On Wed, Nov 4, 2020 at 1:39 AM Steven Wu <
> > > >> stevenz3wu@gmail.com
> > > >> > > >>>
> > > >> > > >>>>> wrote:
> > > >> > > >>>>>>>>
> > > >> > > >>>>>>>>> @Stephan Ewen <se...@apache.org> yeah, we can do
> that.
> > > >> don't
> > > >> > > >>>> worry
> > > >> > > >>>>>>> about
> > > >> > > >>>>>>>>> it. your earlier email had the perfect explanation on
> > why
> > > >> file
> > > >> > > >>>>> source
> > > >> > > >>>>>>>>> shouldn't be backported.
> > > >> > > >>>>>>>>>
> > > >> > > >>>>>>>>> On Tue, Nov 3, 2020 at 3:37 AM Stephan Ewen <
> > > >> sewen@apache.org
> > > >> > > >>>
> > > >> > > >>>>> wrote:
> > > >> > > >>>>>>>>>
> > > >> > > >>>>>>>>>> @Steven would it be possible to initially copy some
> of
> > > the
> > > >> > > >>> code
> > > >> > > >>>>> into
> > > >> > > >>>>>>> the
> > > >> > > >>>>>>>>>> iceberg source and later replace it by a dependency
> on
> > > the
> > > >> > > >>> Flink
> > > >> > > >>>>>> file
> > > >> > > >>>>>>>>>> source?
> > > >> > > >>>>>>>>>>
> > > >> > > >>>>>>>>>> On Mon, Nov 2, 2020 at 8:33 PM Steven Wu <
> > > >> > > >>> stevenz3wu@gmail.com>
> > > >> > > >>>>>>> wrote:
> > > >> > > >>>>>>>>>>
> > > >> > > >>>>>>>>>>> Stephan, thanks a lot for explaining the file
> > connector.
> > > >> > > >>> that
> > > >> > > >>>>>> makes
> > > >> > > >>>>>>>>>> sense.
> > > >> > > >>>>>>>>>>>
> > > >> > > >>>>>>>>>>> I was asking because we were trying to reuse some of
> > the
> > > >> > > >>>>>>>>> implementations
> > > >> > > >>>>>>>>>> in
> > > >> > > >>>>>>>>>>> the file source for Iceberg source. Flink Iceberg
> > source
> > > >> > > >>> lives
> > > >> > > >>>>> in
> > > >> > > >>>>>>> the
> > > >> > > >>>>>>>>>>> Iceberg repo, which is not possible to code against
> > the
> > > >> > > >>> master
> > > >> > > >>>>>>> branch
> > > >> > > >>>>>>>>> of
> > > >> > > >>>>>>>>>>> the Flink code.
> > > >> > > >>>>>>>>>>>
> > > >> > > >>>>>>>>>>> On Mon, Nov 2, 2020 at 3:31 AM Stephan Ewen <
> > > >> > > >>> sewen@apache.org
> > > >> > > >>>>>
> > > >> > > >>>>>>> wrote:
> > > >> > > >>>>>>>>>>>
> > > >> > > >>>>>>>>>>>> Hi Steven!
> > > >> > > >>>>>>>>>>>>
> > > >> > > >>>>>>>>>>>> So far there are no plans to pick back the file
> > system
> > > >> > > >>>>> connector
> > > >> > > >>>>>>>>> code.
> > > >> > > >>>>>>>>>>> This
> > > >> > > >>>>>>>>>>>> is still evolving and not finalized for 1.12, so I
> > > don't
> > > >> > > >>>> feel
> > > >> > > >>>>> it
> > > >> > > >>>>>>> is
> > > >> > > >>>>>>>>> a
> > > >> > > >>>>>>>>>>> good
> > > >> > > >>>>>>>>>>>> candidate to be backported.
> > > >> > > >>>>>>>>>>>> However, with the base connector changes
> backported,
> > > you
> > > >> > > >>>>> should
> > > >> > > >>>>>> be
> > > >> > > >>>>>>>>> able
> > > >> > > >>>>>>>>>>> to
> > > >> > > >>>>>>>>>>>> run the file connector code from master against
> > 1.11.3.
> > > >> > > >>>>>>>>>>>>
> > > >> > > >>>>>>>>>>>> The collect() utils can be picked back, I see no
> > issue
> > > >> > > >>> with
> > > >> > > >>>>> that
> > > >> > > >>>>>>>>> (it is
> > > >> > > >>>>>>>>>>>> isolated utilities).
> > > >> > > >>>>>>>>>>>>
> > > >> > > >>>>>>>>>>>> Best,
> > > >> > > >>>>>>>>>>>> Stephan
> > > >> > > >>>>>>>>>>>>
> > > >> > > >>>>>>>>>>>>
> > > >> > > >>>>>>>>>>>> On Mon, Nov 2, 2020 at 3:02 AM Steven Wu <
> > > >> > > >>>>> stevenz3wu@gmail.com>
> > > >> > > >>>>>>>>> wrote:
> > > >> > > >>>>>>>>>>>>
> > > >> > > >>>>>>>>>>>>> Basically, it would be great to get the latest
> code
> > in
> > > >> > > >>> the
> > > >> > > >>>>>>>>>>>>> flink-connector-files (FLIP-27).
> > > >> > > >>>>>>>>>>>>>
> > > >> > > >>>>>>>>>>>>> On Sat, Oct 31, 2020 at 9:57 AM Steven Wu <
> > > >> > > >>>>>> stevenz3wu@gmail.com
> > > >> > > >>>>>>>>
> > > >> > > >>>>>>>>>>> wrote:
> > > >> > > >>>>>>>>>>>>>
> > > >> > > >>>>>>>>>>>>>> Stephan, it will be great if we can also backport
> > > >> > > >> the
> > > >> > > >>>>>>>>>> DataStreamUtils
> > > >> > > >>>>>>>>>>>>>> related commits that help with collecting output
> > > >> > > >> from
> > > >> > > >>>>>>> unbounded
> > > >> > > >>>>>>>>>>>> streams.
> > > >> > > >>>>>>>>>>>>>> e.g.
> > > >> > > >>>>>>>>>>>>>>
> > > >> > > >>>>>>>>>>>>>>
> > > >> > > >>>>>>>>>>>>>>
> > > >> > > >>>>>>>>>>>>>
> > > >> > > >>>>>>>>>>>>
> > > >> > > >>>>>>>>>>>
> > > >> > > >>>>>>>>>>
> > > >> > > >>>>>>>>>
> > > >> > > >>>>>>>
> > > >> > > >>>>>>
> > > >> > > >>>>>
> > > >> > > >>>>
> > > >> > > >>>
> > > >> > > >>
> > > >> > >
> > > >> >
> > > >>
> > >
> >
> https://github.com/apache/flink/commit/09a7a66b7313fea64817fe960a8da1265b428efc
> > > >> > > >>>>>>>>>>>>>>
> > > >> > > >>>>>>>>>>>>>> I tried to copy and paste the code to unblock
> > > >> > > >> myself.
> > > >> > > >>>> but
> > > >> > > >>>>> it
> > > >> > > >>>>>>>>>> quickly
> > > >> > > >>>>>>>>>>>> got
> > > >> > > >>>>>>>>>>>>>> into the rabbit hole of more and more code.
> > > >> > > >>>>>>>>>>>>>>
> > > >> > > >>>>>>>>>>>>>> On Fri, Oct 30, 2020 at 11:02 AM Stephan Ewen <
> > > >> > > >>>>>>> sewen@apache.org
> > > >> > > >>>>>>>>>>
> > > >> > > >>>>>>>>>>>> wrote:
> > > >> > > >>>>>>>>>>>>>>
> > > >> > > >>>>>>>>>>>>>>> I have started with backporting the source API
> > > >> > > >>> changes.
> > > >> > > >>>>>> Some
> > > >> > > >>>>>>>>> minor
> > > >> > > >>>>>>>>>>>>>>> conflicts to solve, will need a bit more to
> finish
> > > >> > > >>>> this.
> > > >> > > >>>>>>>>>>>>>>>
> > > >> > > >>>>>>>>>>>>>>> On Fri, Oct 30, 2020 at 7:25 AM Tzu-Li (Gordon)
> > > >> > > >> Tai <
> > > >> > > >>>>>>>>>>>>> tzulitai@apache.org>
> > > >> > > >>>>>>>>>>>>>>> wrote:
> > > >> > > >>>>>>>>>>>>>>>
> > > >> > > >>>>>>>>>>>>>>>> @Stephan Ewen <se...@apache.org>
> > > >> > > >>>>>>>>>>>>>>>> Are there already plans or ongoing efforts for
> > > >> > > >>>>>> backporting
> > > >> > > >>>>>>>>> the
> > > >> > > >>>>>>>>>>> list
> > > >> > > >>>>>>>>>>>> of
> > > >> > > >>>>>>>>>>>>>>>> FLIP-27 changes that you posted?
> > > >> > > >>>>>>>>>>>>>>>>
> > > >> > > >>>>>>>>>>>>>>>> On Thu, Oct 29, 2020 at 7:08 PM Xintong Song <
> > > >> > > >>>>>>>>>>> tonysong820@gmail.com
> > > >> > > >>>>>>>>>>>>>
> > > >> > > >>>>>>>>>>>>>>>> wrote:
> > > >> > > >>>>>>>>>>>>>>>>
> > > >> > > >>>>>>>>>>>>>>>>> Hi folks,
> > > >> > > >>>>>>>>>>>>>>>>>
> > > >> > > >>>>>>>>>>>>>>>>> Just to provide some updates concerning the
> > > >> > > >> status
> > > >> > > >>>> on
> > > >> > > >>>>>> the
> > > >> > > >>>>>>>>>>>>>>>>> test instabilities.
> > > >> > > >>>>>>>>>>>>>>>>>
> > > >> > > >>>>>>>>>>>>>>>>> Currently, we have 30 unresolved tickets
> labeled
> > > >> > > >>>> with
> > > >> > > >>>>>>>>> `Affects
> > > >> > > >>>>>>>>>>>>> Version`
> > > >> > > >>>>>>>>>>>>>>>>> 1.11.x.
> > > >> > > >>>>>>>>>>>>>>>>>
> > > >> > > >>>>>>>>>>>>>>>>>
> > > >> > > >>>>>>>>>>>>>>>
> > > >> > > >>>>>>>>>>>>>
> > > >> > > >>>>>>>>>>>>
> > > >> > > >>>>>>>>>>>
> > > >> > > >>>>>>>>>>
> > > >> > > >>>>>>>>>
> > > >> > > >>>>>>>
> > > >> > > >>>>>>
> > > >> > > >>>>>
> > > >> > > >>>>
> > > >> > > >>>
> > > >> > > >>
> > > >> > >
> > > >> >
> > > >>
> > >
> >
> https://issues.apache.org/jira/browse/FLINK-19775?filter=12348580&jql=project%20%3D%20FLINK%20AND%20resolution%20%3D%20Unresolved%20AND%20affectedVersion%20in%20(1.11.0%2C%201.11.1%2C%201.11.2%2C%201.11.3)%20AND%20labels%20%3D%20test-stability%20ORDER%20BY%20created%20DESC
> > > >> > > >>>>>>>>>>>>>>>>>
> > > >> > > >>>>>>>>>>>>>>>>> Among the 30 tickets, 11 of them are:
> > > >> > > >>>>>>>>>>>>>>>>> - Have occured in the recent 3 months
> > > >> > > >>>>>>>>>>>>>>>>> - Not confirmed to be pure testability issues
> > > >> > > >>>>>>>>>>>>>>>>> - Not confirmed to be rare condition cases
> > > >> > > >>>>>>>>>>>>>>>>>
> > > >> > > >>>>>>>>>>>>>>>>> It would be nice if someone familiar with
> these
> > > >> > > >>>>>> components
> > > >> > > >>>>>>>>> can
> > > >> > > >>>>>>>>>>>> take a
> > > >> > > >>>>>>>>>>>>>>> look
> > > >> > > >>>>>>>>>>>>>>>>> into these issues.
> > > >> > > >>>>>>>>>>>>>>>>>
> > > >> > > >>>>>>>>>>>>>>>>> -
> > > >> > > >>> https://issues.apache.org/jira/browse/FLINK-17159
> > > >> > > >>>>>> (ES6)
> > > >> > > >>>>>>>>>>>>>>>>> -
> > > >> > > >>> https://issues.apache.org/jira/browse/FLINK-17912
> > > >> > > >>>>>>> (Kafka)
> > > >> > > >>>>>>>>>>>>>>>>> -
> > > >> > > >>> https://issues.apache.org/jira/browse/FLINK-17949
> > > >> > > >>>>>>> (Kafka)
> > > >> > > >>>>>>>>>>>>>>>>> ⁃
> > > >> > > >>> https://issues.apache.org/jira/browse/FLINK-18444
> > > >> > > >>>>>>> (Kafka)
> > > >> > > >>>>>>>>>>>>>>>>> -
> > > >> > > >>> https://issues.apache.org/jira/browse/FLINK-18634
> > > >> > > >>>>>>> (Kafka)
> > > >> > > >>>>>>>>>>>>>>>>> -
> > > >> > > >>> https://issues.apache.org/jira/browse/FLINK-18648
> > > >> > > >>>>>>> (Kafka)
> > > >> > > >>>>>>>>>>>>>>>>> -
> > > >> > > >>> https://issues.apache.org/jira/browse/FLINK-18807
> > > >> > > >>>>>>> (Kafka)
> > > >> > > >>>>>>>>>>>>>>>>> -
> > > >> > > >>> https://issues.apache.org/jira/browse/FLINK-19369
> > > >> > > >>>>>>>>>>>> (BlobClientTest)
> > > >> > > >>>>>>>>>>>>>>>>> -
> > > >> > > >>> https://issues.apache.org/jira/browse/FLINK-19436
> > > >> > > >>>>>>> (TPCDS)
> > > >> > > >>>>>>>>>>>>>>>>> -
> > > >> > > >>> https://issues.apache.org/jira/browse/FLINK-19690
> > > >> > > >>>>>>>>>>>> (Format/Parquet)
> > > >> > > >>>>>>>>>>>>>>>>> -
> > > >> > > >>> https://issues.apache.org/jira/browse/FLINK-19775
> > > >> > > >>>>>>>>>>>>>>>>> (SystemProcessingTimeServiceTest)
> > > >> > > >>>>>>>>>>>>>>>>>
> > > >> > > >>>>>>>>>>>>>>>>> Thank you~
> > > >> > > >>>>>>>>>>>>>>>>>
> > > >> > > >>>>>>>>>>>>>>>>> Xintong Song
> > > >> > > >>>>>>>>>>>>>>>>>
> > > >> > > >>>>>>>>>>>>>>>>>
> > > >> > > >>>>>>>>>>>>>>>>>
> > > >> > > >>>>>>>>>>>>>>>>> On Thu, Oct 29, 2020 at 10:21 AM Jingsong Li <
> > > >> > > >>>>>>>>>>>> jingsonglee0@gmail.com
> > > >> > > >>>>>>>>>>>>>>
> > > >> > > >>>>>>>>>>>>>>>>> wrote:
> > > >> > > >>>>>>>>>>>>>>>>>
> > > >> > > >>>>>>>>>>>>>>>>>> +1 to backport the FLIP-27 adjustments to
> > > >> > > >>> 1.11.x.
> > > >> > > >>>>>>>>>>>>>>>>>>
> > > >> > > >>>>>>>>>>>>>>>>>> If possible, that would be great. Many people
> > > >> > > >>> are
> > > >> > > >>>>>>> looking
> > > >> > > >>>>>>>>>>> forward
> > > >> > > >>>>>>>>>>>>> to
> > > >> > > >>>>>>>>>>>>>>> the
> > > >> > > >>>>>>>>>>>>>>>>>> FLIP-27 interface, but they don't want to
> take
> > > >> > > >>> the
> > > >> > > >>>>>> risk
> > > >> > > >>>>>>> to
> > > >> > > >>>>>>>>>>>> upgrade
> > > >> > > >>>>>>>>>>>>> to
> > > >> > > >>>>>>>>>>>>>>>>> 1.12
> > > >> > > >>>>>>>>>>>>>>>>>> (And wait 1.12). After all, 1.11 is a
> > > >> > > >> relatively
> > > >> > > >>>>>> stable
> > > >> > > >>>>>>>>>>> version.
> > > >> > > >>>>>>>>>>>>>>>>>>
> > > >> > > >>>>>>>>>>>>>>>>>> Best,
> > > >> > > >>>>>>>>>>>>>>>>>> Jingsong
> > > >> > > >>>>>>>>>>>>>>>>>>
> > > >> > > >>>>>>>>>>>>>>>>>> On Thu, Oct 29, 2020 at 1:24 AM Stephan Ewen
> <
> > > >> > > >>>>>>>>>> sewen@apache.org
> > > >> > > >>>>>>>>>>>>
> > > >> > > >>>>>>>>>>>>>>> wrote:
> > > >> > > >>>>>>>>>>>>>>>>>>
> > > >> > > >>>>>>>>>>>>>>>>>>> Thanks for starting this.
> > > >> > > >>>>>>>>>>>>>>>>>>>
> > > >> > > >>>>>>>>>>>>>>>>>>> +1 form my side to backport the FLIP-27
> > > >> > > >>>>> adjustments
> > > >> > > >>>>>> to
> > > >> > > >>>>>>>>>>> 1.11.x.
> > > >> > > >>>>>>>>>>>>>>>>>>>
> > > >> > > >>>>>>>>>>>>>>>>>>> There were quite a few changes, and I think
> > > >> > > >> we
> > > >> > > >>>>> need
> > > >> > > >>>>>> to
> > > >> > > >>>>>>>>>>>>> cherry-pick
> > > >> > > >>>>>>>>>>>>>>>>> them
> > > >> > > >>>>>>>>>>>>>>>>>> all
> > > >> > > >>>>>>>>>>>>>>>>>>> to not get some inconsistent mix of changes
> > > >> > > >>> and
> > > >> > > >>>>> many
> > > >> > > >>>>>>>>> merge
> > > >> > > >>>>>>>>>>>>>>> conflicts.
> > > >> > > >>>>>>>>>>>>>>>>>>> I made a list below of what we need to add
> > > >> > > >> to
> > > >> > > >>>>>>>>>> "release-1.11".
> > > >> > > >>>>>>>>>>>>>>>>>>>
> > > >> > > >>>>>>>>>>>>>>>>>>> *#### Core Source API Changes to backport
> > > >> > > >> (in
> > > >> > > >>>>>> REVERSE
> > > >> > > >>>>>>>>>> order)*
> > > >> > > >>>>>>>>>>>>>>>>>>>
> > > >> > > >>>>>>>>>>>>>>>>>>>  (Use: "git log
> > > >> > > >>>>>>>>>>>>>>>>>>>
> > > >> > > >>>>>>>>>>>>
> > > >> > > >>>>>
> > > flink-core/src/main/java/org/apache/flink/api/connector/source")
> > > >> > > >>>>>>>>>>>>>>>>>>>
> > > >> > > >>>>>>>>>>>>>>>>>>> commit
> > > >> > > >>> 162c072e9265a7b6dd9d6f5459eb7974091c4c4e
> > > >> > > >>>>>>>>>>>>>>>>>>> [FLINK-19492][core] Consolidate Source
> > > >> > > >> Events
> > > >> > > >>>>>> between
> > > >> > > >>>>>>>>>> Source
> > > >> > > >>>>>>>>>>>> API
> > > >> > > >>>>>>>>>>>>>>> and
> > > >> > > >>>>>>>>>>>>>>>>>> Split
> > > >> > > >>>>>>>>>>>>>>>>>>> Reader API
> > > >> > > >>>>>>>>>>>>>>>>>>>
> > > >> > > >>>>>>>>>>>>>>>>>>> commit
> > > >> > > >>> 428fbbad85379f71a9fbf1a1ab3e3dcfa9006436
> > > >> > > >>>>>>>>>>>>>>>>>>> [FLINK-19205][core] Add access to
> > > >> > > >>> configuration
> > > >> > > >>>>> and
> > > >> > > >>>>>>>>>> hostname
> > > >> > > >>>>>>>>>>> in
> > > >> > > >>>>>>>>>>>>> the
> > > >> > > >>>>>>>>>>>>>>>>>>> SourceReaderContext
> > > >> > > >>>>>>>>>>>>>>>>>>>
> > > >> > > >>>>>>>>>>>>>>>>>>> commit
> > > >> > > >>> 723e1790744ddc78d19c7c978442af1383f38d33
> > > >> > > >>>>>>>>>>>>>>>>>>> [hotfix][core] Add to Source Enumerator
> > > >> > > >>>>> convenience
> > > >> > > >>>>>>>>> methods
> > > >> > > >>>>>>>>>>> to
> > > >> > > >>>>>>>>>>>>>>> assign
> > > >> > > >>>>>>>>>>>>>>>>>>> single split
> > > >> > > >>>>>>>>>>>>>>>>>>>
> > > >> > > >>>>>>>>>>>>>>>>>>> *#### Connector Base Changes to Backport (in
> > > >> > > >>>>> REVERSE
> > > >> > > >>>>>>>>>> order)*
> > > >> > > >>>>>>>>>>>>>>>>>>>
> > > >> > > >>>>>>>>>>>>>>>>>>>  (Use: "git log
> > > >> > > >>>>>>> flink-connectors/flink-connector-base")
> > > >> > > >>>>>>>>>>>>>>>>>>>
> > > >> > > >>>>>>>>>>>>>>>>>>> commit
> > > >> > > >>> 401f56fe9d6b0271260edf9787cdcbfe4d03874d
> > > >> > > >>>>>>>>>>>>>>>>>>> [FLINK-19427][FLINK-19489][tests] Fix test
> > > >> > > >>>>>> conditions
> > > >> > > >>>>>>>>> for
> > > >> > > >>>>>>>>>>>>>>>>>>>
> > > >> > > >>>>>>> 'SplitFetcherTest.testNotifiesWhenGoingIdleConcurrent()'
> > > >> > > >>>>>>>>>>>>>>>>>>>
> > > >> > > >>>>>>>>>>>>>>>>>>> commit
> > > >> > > >>> 68c5c2ff779d82a1ff81ffaf60d8a1b283797db1
> > > >> > > >>>>>>>>>>>>>>>>>>> [FLINK-19448][connector base] Explicitly
> > > >> > > >> check
> > > >> > > >>>> for
> > > >> > > >>>>>>>>>>> un-expected
> > > >> > > >>>>>>>>>>>>>>>>> condition
> > > >> > > >>>>>>>>>>>>>>>>>>> that would leave an inconsistent state
> > > >> > > >>>>>>>>>>>>>>>>>>>
> > > >> > > >>>>>>>>>>>>>>>>>>> commit
> > > >> > > >>> 162c072e9265a7b6dd9d6f5459eb7974091c4c4e
> > > >> > > >>>>>>>>>>>>>>>>>>> [FLINK-19492][core] Consolidate Source
> > > >> > > >> Events
> > > >> > > >>>>>> between
> > > >> > > >>>>>>>>>> Source
> > > >> > > >>>>>>>>>>>> API
> > > >> > > >>>>>>>>>>>>>>> and
> > > >> > > >>>>>>>>>>>>>>>>>> Split
> > > >> > > >>>>>>>>>>>>>>>>>>> Reader API
> > > >> > > >>>>>>>>>>>>>>>>>>>
> > > >> > > >>>>>>>>>>>>>>>>>>> commit
> > > >> > > >>> c1ca7a4c7c21ec8868c14cdddd43c559625b794c
> > > >> > > >>>>>>>>>>>>>>>>>>> [refactor][tests] Move some source test
> > > >> > > >> utils
> > > >> > > >>>> from
> > > >> > > >>>>>>>>>>>>>>>>> flink-connector-base
> > > >> > > >>>>>>>>>>>>>>>>>> to
> > > >> > > >>>>>>>>>>>>>>>>>>> flink-core
> > > >> > > >>>>>>>>>>>>>>>>>>>
> > > >> > > >>>>>>>>>>>>>>>>>>> commit
> > > >> > > >>> ee5c4c211c35c70d28252363bbc8400453609977
> > > >> > > >>>>>>>>>>>>>>>>>>> [FLINK-19251][connectors] Avoid confusing
> > > >> > > >>> queue
> > > >> > > >>>>>>>>> handling in
> > > >> > > >>>>>>>>>>>>>>>>>>> "SplitReader.handleSplitsChanges()"
> > > >> > > >>>>>>>>>>>>>>>>>>>
> > > >> > > >>>>>>>>>>>>>>>>>>> commit
> > > >> > > >>> 5abef56b2bf85bcac786f6b16b6899b6cced7176
> > > >> > > >>>>>>>>>>>>>>>>>>> [FLINK-19250][connectors] Fix error
> > > >> > > >>> propagation
> > > >> > > >>>> in
> > > >> > > >>>>>>>>>> connector
> > > >> > > >>>>>>>>>>>> base
> > > >> > > >>>>>>>>>>>>>>>>>>> (SplitFetcherManager).
> > > >> > > >>>>>>>>>>>>>>>>>>>
> > > >> > > >>>>>>>>>>>>>>>>>>> commit
> > > >> > > >>> 8fcca837c55a9216595ee4c03038b52747098dbb
> > > >> > > >>>>>>>>>>>>>>>>>>> [hotfix][connectors] Improve JavaDocs for
> > > >> > > >>>>>>>>>>>>>>> SingleThreadFetcherManager
> > > >> > > >>>>>>>>>>>>>>>>>>>
> > > >> > > >>>>>>>>>>>>>>>>>>> commit
> > > >> > > >>> 4700bb5dde3303cbe98882f6beb7379425717b01
> > > >> > > >>>>>>>>>>>>>>>>>>> [FLINK-19225][connectors] Various small
> > > >> > > >>>>> improvements
> > > >> > > >>>>>>> to
> > > >> > > >>>>>>>>>>>>>>>>> SourceReaderBase
> > > >> > > >>>>>>>>>>>>>>>>>>> (part 2)
> > > >> > > >>>>>>>>>>>>>>>>>>>
> > > >> > > >>>>>>>>>>>>>>>>>>> commit
> > > >> > > >>> 12261c6b7ed6478a9b9f6a69cb58246b83cab9b7
> > > >> > > >>>>>>>>>>>>>>>>>>> [FLINK-17393][connectors] (follow-up) Wakeup
> > > >> > > >>> the
> > > >> > > >>>>>>>>>>> SplitFetchers
> > > >> > > >>>>>>>>>>>>> more
> > > >> > > >>>>>>>>>>>>>>>>>>> elegantly.
> > > >> > > >>>>>>>>>>>>>>>>>>>
> > > >> > > >>>>>>>>>>>>>>>>>>> commit
> > > >> > > >>> c60aaff0249bfd6b5871b7f82e03efc487a54d6b
> > > >> > > >>>>>>>>>>>>>>>>>>> [hotfix][tests] Extend test coverage for
> > > >> > > >>>>>>>>>>>>>>> FutureCompletingBlockingQueue
> > > >> > > >>>>>>>>>>>>>>>>>>>
> > > >> > > >>>>>>>>>>>>>>>>>>> commit
> > > >> > > >>> cef8a587d7fd2fe64cc644da5ed095d82e46f631
> > > >> > > >>>>>>>>>>>>>>>>>>> [FLINK-19245][connectors] Set default
> > > >> > > >> capacity
> > > >> > > >>>> for
> > > >> > > >>>>>>>>>>>>>>>>>>> FutureCompletingBlockingQueue.
> > > >> > > >>>>>>>>>>>>>>>>>>>
> > > >> > > >>>>>>>>>>>>>>>>>>> commit
> > > >> > > >>> 4ea95782b4c6a2538153d4d16ad3f4839c7de0fb
> > > >> > > >>>>>>>>>>>>>>>>>>> [FLINK-19223][connectors] Simplify
> > > >> > > >>> Availability
> > > >> > > >>>>>> Future
> > > >> > > >>>>>>>>>> Model
> > > >> > > >>>>>>>>>>> in
> > > >> > > >>>>>>>>>>>>>>> Base
> > > >> > > >>>>>>>>>>>>>>>>>>> Connector
> > > >> > > >>>>>>>>>>>>>>>>>>>
> > > >> > > >>>>>>>>>>>>>>>>>>> commit
> > > >> > > >>> 511857049ba30c8ff0ee56da551fa4a479dc583e
> > > >> > > >>>>>>>>>>>>>>>>>>> [FLINK-18128][connectors] Ensure idle split
> > > >> > > >>>>> fetchers
> > > >> > > >>>>>>>>> lead
> > > >> > > >>>>>>>>>> to
> > > >> > > >>>>>>>>>>>>>>>>> availability
> > > >> > > >>>>>>>>>>>>>>>>>>> notifications.
> > > >> > > >>>>>>>>>>>>>>>>>>>
> > > >> > > >>>>>>>>>>>>>>>>>>> commit
> > > >> > > >>> a8206467af0830dcb89623ea068b5ca3b3450c92
> > > >> > > >>>>>>>>>>>>>>>>>>> [refactor][core] Eagerly initialize the
> > > >> > > >>>> FetchTask
> > > >> > > >>>>> to
> > > >> > > >>>>>>>>>> support
> > > >> > > >>>>>>>>>>>>> proper
> > > >> > > >>>>>>>>>>>>>>>>> unit
> > > >> > > >>>>>>>>>>>>>>>>>>> testing
> > > >> > > >>>>>>>>>>>>>>>>>>>
> > > >> > > >>>>>>>>>>>>>>>>>>> commit
> > > >> > > >>> 3b2f54bcb437f98e6137c904045cc51072b5c06b
> > > >> > > >>>>>>>>>>>>>>>>>>> [hotfix][tests] Move constants in
> > > >> > > >>>> SplitFetcherTest
> > > >> > > >>>>>>>>> relevant
> > > >> > > >>>>>>>>>>> to
> > > >> > > >>>>>>>>>>>>> only
> > > >> > > >>>>>>>>>>>>>>>>> one
> > > >> > > >>>>>>>>>>>>>>>>>>> test into test method
> > > >> > > >>>>>>>>>>>>>>>>>>>
> > > >> > > >>>>>>>>>>>>>>>>>>> commit
> > > >> > > >>> d7625760a75a508bf05bcddc380bb4d62ee1743e
> > > >> > > >>>>>>>>>>>>>>>>>>> [FLINK-19225][connectors] Various small
> > > >> > > >>>>> improvements
> > > >> > > >>>>>>> to
> > > >> > > >>>>>>>>>>>>>>>>> SourceReaderBase
> > > >> > > >>>>>>>>>>>>>>>>>>>
> > > >> > > >>>>>>>>>>>>>>>>>>> commit
> > > >> > > >>> a5b0d3297748c1be47ad579a88f24df2255a8df1
> > > >> > > >>>>>>>>>>>>>>>>>>> [FLINK-17393][connectors] Wakeup the
> > > >> > > >>>> SplitFetchers
> > > >> > > >>>>>>> more
> > > >> > > >>>>>>>>>>>>> elegantly.
> > > >> > > >>>>>>>>>>>>>>>>>>>
> > > >> > > >>>>>>>>>>>>>>>>>>> commit
> > > >> > > >>> f42a3ebc3e81a034b7221a803c153636fef34903
> > > >> > > >>>>>>>>>>>>>>>>>>> [FLINK-18680][connectors] Make connector
> > > >> > > >> base
> > > >> > > >>>>>>>>>>>> RecordsWithSplitIds
> > > >> > > >>>>>>>>>>>>>>> more
> > > >> > > >>>>>>>>>>>>>>>>>>> lightweight.
> > > >> > > >>>>>>>>>>>>>>>>>>>
> > > >> > > >>>>>>>>>>>>>>>>>>> commit
> > > >> > > >>> e3d273de822b085183d09b275a445879ff94b350
> > > >> > > >>>>>>>>>>>>>>>>>>> [FLINK-19162][connectors] Add 'recycle()' to
> > > >> > > >>> the
> > > >> > > >>>>>>>>>>>>>>> RecordsWithSplitIds
> > > >> > > >>>>>>>>>>>>>>>>> to
> > > >> > > >>>>>>>>>>>>>>>>>>> support reuse of heavy objects.
> > > >> > > >>>>>>>>>>>>>>>>>>>
> > > >> > > >>>>>>>>>>>>>>>>>>> commit
> > > >> > > >>> 8ebc464c2520453a70001cd712abc8dee6ee89e0
> > > >> > > >>>>>>>>>>>>>>>>>>> [hotfix][testing] Add a set of
> > > >> > > >> parameterizable
> > > >> > > >>>>>> testing
> > > >> > > >>>>>>>>>> mocks
> > > >> > > >>>>>>>>>>>> for
> > > >> > > >>>>>>>>>>>>>>> the
> > > >> > > >>>>>>>>>>>>>>>>>> Split
> > > >> > > >>>>>>>>>>>>>>>>>>> Reader API
> > > >> > > >>>>>>>>>>>>>>>>>>>
> > > >> > > >>>>>>>>>>>>>>>>>>> commit
> > > >> > > >>> 930a07438be1185388d7150640f294dfe2a5d378
> > > >> > > >>>>>>>>>>>>>>>>>>> [hotfix][connectors] Add RequestSplitEvent
> > > >> > > >> to
> > > >> > > >>>>>>>>>>>>>>> 'flink-connector-base'
> > > >> > > >>>>>>>>>>>>>>>>>>>
> > > >> > > >>>>>>>>>>>>>>>>>>> commit
> > > >> > > >>> e945ce8a933bc378844782f784ca473c767ca159
> > > >> > > >>>>>>>>>>>>>>>>>>> [refactor][connectors] Backport of the
> > > >> > > >>>>>> connector-base
> > > >> > > >>>>>>>>>>> exception
> > > >> > > >>>>>>>>>>>>>>>>> handling
> > > >> > > >>>>>>>>>>>>>>>>>>> from the Kafka Connector Pull Request
> > > >> > > >>>>>>>>>>>>>>>>>>>
> > > >> > > >>>>>>>>>>>>>>>>>>>
> > > >> > > >>>>>>>>>>>>>>>>>>> On Wed, Oct 28, 2020 at 5:49 PM Steven Wu <
> > > >> > > >>>>>>>>>>>> stevenz3wu@gmail.com>
> > > >> > > >>>>>>>>>>>>>>>>> wrote:
> > > >> > > >>>>>>>>>>>>>>>>>>>
> > > >> > > >>>>>>>>>>>>>>>>>>>> I would love to see this FLIP-27 source
> > > >> > > >>>>> interface
> > > >> > > >>>>>>>>>>> improvement
> > > >> > > >>>>>>>>>>>>> [1]
> > > >> > > >>>>>>>>>>>>>>>>> made
> > > >> > > >>>>>>>>>>>>>>>>>> to
> > > >> > > >>>>>>>>>>>>>>>>>>>> 1.11.3.
> > > >> > > >>>>>>>>>>>>>>>>>>>>
> > > >> > > >>>>>>>>>>>>>>>>>>>> [1]
> > > >> > > >>>>>>> https://issues.apache.org/jira/browse/FLINK-19698
> > > >> > > >>>>>>>>>>>>>>>>>>>>
> > > >> > > >>>>>>>>>>>>>>>>>>>> On Wed, Oct 28, 2020 at 12:32 AM Tzu-Li
> > > >> > > >>>> (Gordon)
> > > >> > > >>>>>>> Tai <
> > > >> > > >>>>>>>>>>>>>>>>>>> tzulitai@apache.org>
> > > >> > > >>>>>>>>>>>>>>>>>>>> wrote:
> > > >> > > >>>>>>>>>>>>>>>>>>>>
> > > >> > > >>>>>>>>>>>>>>>>>>>>> Thanks for the replies so far!
> > > >> > > >>>>>>>>>>>>>>>>>>>>>
> > > >> > > >>>>>>>>>>>>>>>>>>>>> Just to provide a brief update on the
> > > >> > > >>> status
> > > >> > > >>>>> of
> > > >> > > >>>>>>>>>> blockers
> > > >> > > >>>>>>>>>>>> for
> > > >> > > >>>>>>>>>>>>>>>>> 1.11.3
> > > >> > > >>>>>>>>>>>>>>>>>> so
> > > >> > > >>>>>>>>>>>>>>>>>>>> far:
> > > >> > > >>>>>>>>>>>>>>>>>>>>>
> > > >> > > >>>>>>>>>>>>>>>>>>>>>
> > > >> > > >>>>>>>>>>>>>>>>>>>>> *PR opened, pending reviewer*-
> > > >> > > >>> [FLINK-19717]
> > > >> > > >>>>>>>>>>>>>>>>>> SourceReaderBase.pollNext
> > > >> > > >>>>>>>>>>>>>>>>>>>> may
> > > >> > > >>>>>>>>>>>>>>>>>>>>> return END_OF_INPUT if SplitReader.fetch
> > > >> > > >>>>> throws
> > > >> > > >>>>>> (
> > > >> > > >>>>>>>>>>>>>>>>>>>>>
> > > >> > > >>> https://github.com/apache/flink/pull/13776)
> > > >> > > >>>>>>>>>>>>>>>>>>>>>
> > > >> > > >>>>>>>>>>>>>>>>>>>>> *PR opened, reviewed + close to being
> > > >> > > >>>> merged*
> > > >> > > >>>>>>>>>>>>>>>>>>>>> - [FLINK-19741] Timer service should
> > > >> > > >> skip
> > > >> > > >>>>>>> restoring
> > > >> > > >>>>>>>>>> from
> > > >> > > >>>>>>>>>>>> raw
> > > >> > > >>>>>>>>>>>>>>> keyed
> > > >> > > >>>>>>>>>>>>>>>>>>> stream
> > > >> > > >>>>>>>>>>>>>>>>>>>>> if it isn't the writer (
> > > >> > > >>>>>>>>>>>>>>>>> https://github.com/apache/flink/pull/13761)
> > > >> > > >>>>>>>>>>>>>>>>>>>>> - [FLINK-19748] Raw keyed stream key
> > > >> > > >> group
> > > >> > > >>>>>>> iterator
> > > >> > > >>>>>>>>>>> should
> > > >> > > >>>>>>>>>>>> be
> > > >> > > >>>>>>>>>>>>>>>>>> skipping
> > > >> > > >>>>>>>>>>>>>>>>>>>>> unwritten key groups (
> > > >> > > >>>>>>>>>>>>>>> https://github.com/apache/flink/pull/13772)
> > > >> > > >>>>>>>>>>>>>>>>>>>>>
> > > >> > > >>>>>>>>>>>>>>>>>>>>> *Merged*
> > > >> > > >>>>>>>>>>>>>>>>>>>>> - [FLINK-19154] Application mode deletes
> > > >> > > >>> HA
> > > >> > > >>>>> data
> > > >> > > >>>>>>> in
> > > >> > > >>>>>>>>>> case
> > > >> > > >>>>>>>>>>> of
> > > >> > > >>>>>>>>>>>>>>>>> suspended
> > > >> > > >>>>>>>>>>>>>>>>>>>>> ZooKeeper connection
> > > >> > > >>>>>>>>>>>>>>>>>>>>> - [FLINK-19569] Upgrade ICU4J to 67.1+
> > > >> > > >>>>>>>>>>>>>>>>>>>>>
> > > >> > > >>>>>>>>>>>>>>>>>>>>> Right now as it seems, progress is
> > > >> > > >> mainly
> > > >> > > >>>>>> blocked
> > > >> > > >>>>>>>>> on a
> > > >> > > >>>>>>>>>>>>> reviewer
> > > >> > > >>>>>>>>>>>>>>>>> for
> > > >> > > >>>>>>>>>>>>>>>>>>>>> FLINK-19717.
> > > >> > > >>>>>>>>>>>>>>>>>>>>> Meanwhile, Xintong is keeping an eye on
> > > >> > > >>> test
> > > >> > > >>>>>>>>>>> instabilities
> > > >> > > >>>>>>>>>>>>> [1]
> > > >> > > >>>>>>>>>>>>>>> to
> > > >> > > >>>>>>>>>>>>>>>>> see
> > > >> > > >>>>>>>>>>>>>>>>>>> if
> > > >> > > >>>>>>>>>>>>>>>>>>>>> there are any fixes that should be
> > > >> > > >> applied
> > > >> > > >>>> to
> > > >> > > >>>>>>>>>>>> `release-1.11`.
> > > >> > > >>>>>>>>>>>>>>>>>>>>>
> > > >> > > >>>>>>>>>>>>>>>>>>>>> This is also a reminder, that if there
> > > >> > > >> are
> > > >> > > >>>>> other
> > > >> > > >>>>>>>>>> blockers
> > > >> > > >>>>>>>>>>>>> that
> > > >> > > >>>>>>>>>>>>>>> we
> > > >> > > >>>>>>>>>>>>>>>>>> need
> > > >> > > >>>>>>>>>>>>>>>>>>> to
> > > >> > > >>>>>>>>>>>>>>>>>>>>> be aware of, or a need to re-establish
> > > >> > > >>>>> estimated
> > > >> > > >>>>>>>>> time
> > > >> > > >>>>>>>>>> for
> > > >> > > >>>>>>>>>>>>>>> getting
> > > >> > > >>>>>>>>>>>>>>>>>> fixes
> > > >> > > >>>>>>>>>>>>>>>>>>>> in
> > > >> > > >>>>>>>>>>>>>>>>>>>>> and delay the RC for 1.11.3, please do
> > > >> > > >> let
> > > >> > > >>>> us
> > > >> > > >>>>>>> know!
> > > >> > > >>>>>>>>>>>>>>>>>>>>>
> > > >> > > >>>>>>>>>>>>>>>>>>>>> Cheers,
> > > >> > > >>>>>>>>>>>>>>>>>>>>> Gordon
> > > >> > > >>>>>>>>>>>>>>>>>>>>>
> > > >> > > >>>>>>>>>>>>>>>>>>>>> [1]
> > > >> > > >>>>>>>>>>>>>>>>>>>>>
> > > >> > > >>>>>>>>>>>>>>>>>>>>>
> > > >> > > >>>>>>>>>>>>>>>>>>>>
> > > >> > > >>>>>>>>>>>>>>>>>>>
> > > >> > > >>>>>>>>>>>>>>>>>>
> > > >> > > >>>>>>>>>>>>>>>>>
> > > >> > > >>>>>>>>>>>>>>>
> > > >> > > >>>>>>>>>>>>>
> > > >> > > >>>>>>>>>>>>
> > > >> > > >>>>>>>>>>>
> > > >> > > >>>>>>>>>>
> > > >> > > >>>>>>>>>
> > > >> > > >>>>>>>
> > > >> > > >>>>>>
> > > >> > > >>>>>
> > > >> > > >>>>
> > > >> > > >>>
> > > >> > > >>
> > > >> > >
> > > >> >
> > > >>
> > >
> >
> https://issues.apache.org/jira/issues/?jql=project%20%3D%20FLINK%20AND%20status%20in%20(Open%2C%20%22In%20Progress%22%2C%20Reopened)%20AND%20affectedVersion%20in%20(1.11.0%2C%201.11.1%2C%201.11.2)%20AND%20labels%20%3D%20test-stability
> > > >> > > >>>>>>>>>>>>>>>>>>>>>
> > > >> > > >>>>>>>>>>>>>>>>>>>>> On Mon, Oct 26, 2020 at 9:43 PM Kostas
> > > >> > > >>>>> Kloudas <
> > > >> > > >>>>>>>>>>>>>>>>> kkloudas@apache.org>
> > > >> > > >>>>>>>>>>>>>>>>>>>>> wrote:
> > > >> > > >>>>>>>>>>>>>>>>>>>>>
> > > >> > > >>>>>>>>>>>>>>>>>>>>>> +1 for releasing Flink 1.11.3 as it
> > > >> > > >>>>> contains a
> > > >> > > >>>>>>>>> number
> > > >> > > >>>>>>>>>>> of
> > > >> > > >>>>>>>>>>>>>>>>> important
> > > >> > > >>>>>>>>>>>>>>>>>>>>>> fixes and thanks Gordon and Xintong
> > > >> > > >> for
> > > >> > > >>>>>>>>> volunteering.
> > > >> > > >>>>>>>>>>>>>>>>>>>>>>
> > > >> > > >>>>>>>>>>>>>>>>>>>>>> Cheers,
> > > >> > > >>>>>>>>>>>>>>>>>>>>>> Kostas
> > > >> > > >>>>>>>>>>>>>>>>>>>>>>
> > > >> > > >>>>>>>>>>>>>>>>>>>>>> On Mon, Oct 26, 2020 at 4:37 AM Yu Li
> > > >> > > >> <
> > > >> > > >>>>>>>>>>> carp84@gmail.com>
> > > >> > > >>>>>>>>>>>>>>> wrote:
> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>
> > > >> > > >>>>>>>>>>>>>>>>>>>>>>> +1 for releasing Flink 1.11.3, and
> > > >> > > >>>> thanks
> > > >> > > >>>>>>> Gordon
> > > >> > > >>>>>>>>>> and
> > > >> > > >>>>>>>>>>>>>>> Xintong
> > > >> > > >>>>>>>>>>>>>>>>> for
> > > >> > > >>>>>>>>>>>>>>>>>>>>>>> volunteering as our release
> > > >> > > >> managers.
> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>
> > > >> > > >>>>>>>>>>>>>>>>>>>>>>> Best Regards,
> > > >> > > >>>>>>>>>>>>>>>>>>>>>>> Yu
> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>
> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>
> > > >> > > >>>>>>>>>>>>>>>>>>>>>>> On Mon, 26 Oct 2020 at 09:45,
> > > >> > > >> Xintong
> > > >> > > >>>>> Song <
> > > >> > > >>>>>>>>>>>>>>>>>> tonysong820@gmail.com>
> > > >> > > >>>>>>>>>>>>>>>>>>>>>> wrote:
> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>
> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>> Thanks Gordan for starting this
> > > >> > > >>>>>> discussion.
> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>> My pleasure to help with the
> > > >> > > >> release
> > > >> > > >>>>>>> process.
> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>
> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>> Thank you~
> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>
> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>> Xintong Song
> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>
> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>
> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>
> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>> On Fri, Oct 23, 2020 at 11:29 PM
> > > >> > > >>> Till
> > > >> > > >>>>>>>>> Rohrmann <
> > > >> > > >>>>>>>>>>>>>>>>>>>> trohrmann@apache.org
> > > >> > > >>>>>>>>>>>>>>>>>>>>>>
> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>> wrote:
> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>
> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>> Thanks for starting this
> > > >> > > >>> discussion
> > > >> > > >>>>>>> Gordon.
> > > >> > > >>>>>>>>>> There
> > > >> > > >>>>>>>>>>>> are
> > > >> > > >>>>>>>>>>>>>>> over
> > > >> > > >>>>>>>>>>>>>>>>>> 100
> > > >> > > >>>>>>>>>>>>>>>>>>>>> issues
> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>> which are fixed for 1.11.3.
> > > >> > > >> Hence
> > > >> > > >>> +1
> > > >> > > >>>>>> for a
> > > >> > > >>>>>>>>>>> soonish
> > > >> > > >>>>>>>>>>>>>>> 1.11.3
> > > >> > > >>>>>>>>>>>>>>>>>>>> release.
> > > >> > > >>>>>>>>>>>>>>>>>>>>>> Thanks
> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>> for volunteering as our release
> > > >> > > >>>>> managers
> > > >> > > >>>>>>>>> Gordon
> > > >> > > >>>>>>>>>>> and
> > > >> > > >>>>>>>>>>>>>>>>> Xintong!
> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>> Cheers,
> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>> Till
> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>> On Fri, Oct 23, 2020 at 5:02 PM
> > > >> > > >>>> Tzu-Li
> > > >> > > >>>>>>>>> (Gordon)
> > > >> > > >>>>>>>>>>>> Tai <
> > > >> > > >>>>>>>>>>>>>>>>>>>>>> tzulitai@apache.org
> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>> wrote:
> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>> Hi,
> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>> Xintong and I would like to
> > > >> > > >>> start a
> > > >> > > >>>>>>>>> discussion
> > > >> > > >>>>>>>>>>> for
> > > >> > > >>>>>>>>>>>>>>>>> releasing
> > > >> > > >>>>>>>>>>>>>>>>>>>> Flink
> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>> 1.11.3
> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>> soon.
> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>> It seems like we already have a
> > > >> > > >>> few
> > > >> > > >>>>>>>>> pressing
> > > >> > > >>>>>>>>>>>> issues
> > > >> > > >>>>>>>>>>>>>>> that
> > > >> > > >>>>>>>>>>>>>>>>>> needs
> > > >> > > >>>>>>>>>>>>>>>>>>>> to
> > > >> > > >>>>>>>>>>>>>>>>>>>>> be
> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>> included in a new hotfix
> > > >> > > >> release:
> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>>   - Heap-based timers’ restore
> > > >> > > >>>>>> behaviour
> > > >> > > >>>>>>>>> is
> > > >> > > >>>>>>>>>>>>> causing a
> > > >> > > >>>>>>>>>>>>>>>>>>> critical
> > > >> > > >>>>>>>>>>>>>>>>>>>>>> recovery
> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>>   issue for StateFun [1] [2]
> > > >> > > >>> [3].
> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>>   - There are several
> > > >> > > >> robustness
> > > >> > > >>>>>> issues
> > > >> > > >>>>>>>>> for
> > > >> > > >>>>>>>>>> the
> > > >> > > >>>>>>>>>>>>>>> FLIP-27
> > > >> > > >>>>>>>>>>>>>>>>> new
> > > >> > > >>>>>>>>>>>>>>>>>>>>> source
> > > >> > > >>>>>>>>>>>>>>>>>>>>>> API,
> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>>   such as [4]. We already have
> > > >> > > >>>> some
> > > >> > > >>>>>>> users
> > > >> > > >>>>>>>>>> using
> > > >> > > >>>>>>>>>>>> the
> > > >> > > >>>>>>>>>>>>>>>>> FLIP-27
> > > >> > > >>>>>>>>>>>>>>>>>>> API
> > > >> > > >>>>>>>>>>>>>>>>>>>>>> with
> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>> 1.11.x,
> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>>   so it would be important to
> > > >> > > >>> get
> > > >> > > >>>>>> those
> > > >> > > >>>>>>>>> fixes
> > > >> > > >>>>>>>>>>> in
> > > >> > > >>>>>>>>>>>>> for
> > > >> > > >>>>>>>>>>>>>>>>> 1.11.x
> > > >> > > >>>>>>>>>>>>>>>>>>> as
> > > >> > > >>>>>>>>>>>>>>>>>>>>>> well.
> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>> Apart from the issues that are
> > > >> > > >>>>> already
> > > >> > > >>>>>>>>> marked
> > > >> > > >>>>>>>>>> as
> > > >> > > >>>>>>>>>>>>>>> blocker
> > > >> > > >>>>>>>>>>>>>>>>> for
> > > >> > > >>>>>>>>>>>>>>>>>>>>> 1.11.3
> > > >> > > >>>>>>>>>>>>>>>>>>>>>> in
> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>> our
> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>> JIRA [5], please let us know in
> > > >> > > >>>> this
> > > >> > > >>>>>>>>> thread if
> > > >> > > >>>>>>>>>>>> there
> > > >> > > >>>>>>>>>>>>>>> is
> > > >> > > >>>>>>>>>>>>>>>>>>> already
> > > >> > > >>>>>>>>>>>>>>>>>>>>>> ongoing
> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>> work for other important fixes
> > > >> > > >>> that
> > > >> > > >>>>> we
> > > >> > > >>>>>>>>> should
> > > >> > > >>>>>>>>>>> try
> > > >> > > >>>>>>>>>>>> to
> > > >> > > >>>>>>>>>>>>>>>>>> include.
> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>> Xintong and I would like to
> > > >> > > >>>> volunteer
> > > >> > > >>>>>> for
> > > >> > > >>>>>>>>>>> managing
> > > >> > > >>>>>>>>>>>>>>> this
> > > >> > > >>>>>>>>>>>>>>>>>>> release,
> > > >> > > >>>>>>>>>>>>>>>>>>>>> and
> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>> will
> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>> try to communicate the priority
> > > >> > > >>> of
> > > >> > > >>>>>>> pending
> > > >> > > >>>>>>>>>>>> blockers
> > > >> > > >>>>>>>>>>>>>>> over
> > > >> > > >>>>>>>>>>>>>>>>> the
> > > >> > > >>>>>>>>>>>>>>>>>>>> next
> > > >> > > >>>>>>>>>>>>>>>>>>>>>> few
> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>> days.
> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>> Since the aforementioned issues
> > > >> > > >>> are
> > > >> > > >>>>>> quite
> > > >> > > >>>>>>>>>>>> critical,
> > > >> > > >>>>>>>>>>>>>>> we’d
> > > >> > > >>>>>>>>>>>>>>>>>> like
> > > >> > > >>>>>>>>>>>>>>>>>>> to
> > > >> > > >>>>>>>>>>>>>>>>>>>>> aim
> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>> for a *feature
> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>> freeze by the end of next week
> > > >> > > >>>> (Oct.
> > > >> > > >>>>>>> 30th)*
> > > >> > > >>>>>>>>>> and
> > > >> > > >>>>>>>>>>>>> start
> > > >> > > >>>>>>>>>>>>>>> the
> > > >> > > >>>>>>>>>>>>>>>>>>>> release
> > > >> > > >>>>>>>>>>>>>>>>>>>>>> voting
> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>> process the week after.
> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>> If that is too short of a
> > > >> > > >> notice
> > > >> > > >>>> and
> > > >> > > >>>>>> you
> > > >> > > >>>>>>>>> might
> > > >> > > >>>>>>>>>>>> need
> > > >> > > >>>>>>>>>>>>>>> more
> > > >> > > >>>>>>>>>>>>>>>>>> time,
> > > >> > > >>>>>>>>>>>>>>>>>>>>>> please
> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>> let
> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>> us know!
> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>> Cheers,
> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>> Gordon
> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>> [1]
> > > >> > > >>>>>>>>>>>>> https://issues.apache.org/jira/browse/FLINK-19692
> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>> [2]
> > > >> > > >>>>>>>>>>>>> https://issues.apache.org/jira/browse/FLINK-19741
> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>> [3]
> > > >> > > >>>>>>>>>>>>> https://issues.apache.org/jira/browse/FLINK-19748
> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>> [4]
> > > >> > > >>>>>>>>>>>>> https://issues.apache.org/jira/browse/FLINK-19717
> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>> [5]
> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>
> > > >> > > >>>>>>>>>>>>>>>>>>>>>>
> > > >> > > >>>>>>>>>>>>>>>>>>>>>
> > > >> > > >>>>>>>>>>>>>>>>>>>>
> > > >> > > >>>>>>>>>>>>>>>>>>>
> > > >> > > >>>>>>>>>>>>>>>>>>
> > > >> > > >>>>>>>>>>>>>>>>>
> > > >> > > >>>>>>>>>>>>>>>
> > > >> > > >>>>>>>>>>>>>
> > > >> > > >>>>>>>>>>>>
> > > >> > > >>>>>>>>>>>
> > > >> > > >>>>>>>>>>
> > > >> > > >>>>>>>>>
> > > >> > > >>>>>>>
> > > >> > > >>>>>>
> > > >> > > >>>>>
> > > >> > > >>>>
> > > >> > > >>>
> > > >> > > >>
> > > >> > >
> > > >> >
> > > >>
> > >
> >
> https://issues.apache.org/jira/issues/?jql=project%20%3D%20FLINK%20AND%20priority%20%3D%20Blocker%20AND%20fixVersion%20%3D%201.11.3
> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > >> > > >>>>>>>>>>>>>>>>>>>>>>>>
> > > >> > > >>>>>>>>>>>>>>>>>>>>>>
> > > >> > > >>>>>>>>>>>>>>>>>>>>>
> > > >> > > >>>>>>>>>>>>>>>>>>>>
> > > >> > > >>>>>>>>>>>>>>>>>>>
> > > >> > > >>>>>>>>>>>>>>>>>>
> > > >> > > >>>>>>>>>>>>>>>>>>
> > > >> > > >>>>>>>>>>>>>>>>>> --
> > > >> > > >>>>>>>>>>>>>>>>>> Best, Jingsong Lee
> > > >> > > >>>>>>>>>>>>>>>>>>
> > > >> > > >>>>>>>>>>>>>>>>>
> > > >> > > >>>>>>>>>>>>>>>>
> > > >> > > >>>>>>>>>>>>>>>
> > > >> > > >>>>>>>>>>>>>>
> > > >> > > >>>>>>>>>>>>>
> > > >> > > >>>>>>>>>>>>
> > > >> > > >>>>>>>>>>>
> > > >> > > >>>>>>>>>>
> > > >> > > >>>>>>>>>
> > > >> > > >>>>>>>>
> > > >> > > >>>>>>>
> > > >> > > >>>>>>
> > > >> > > >>>>>
> > > >> > > >>>>
> > > >> > > >>>
> > > >> > > >>
> > > >> > >
> > > >> > >
> > > >> > >
> > > >> >
> > > >>
> > > >
> > >
> >
>

Re: [DISCUSS] Releasing Apache Flink 1.11.3

Posted by Zhu Zhu <re...@gmail.com>.
Hi Xintong,
We just found an issue FLINK-20285 and I think it is a blocker for 1.11.3.

Thanks,
Zhu

Gyula Fóra <gy...@gmail.com> 于2020年11月18日周三 下午9:36写道：

> Hi All!
>
> I have found the following issue today which might be considered a blocker
> for this release as well:
>
> https://issues.apache.org/jira/browse/FLINK-20221
>
> Could someone please quickly provide a second set of eyes and confirm that
> this is indeed a big problem? :)
>
> Thank you!
> Gyula
>
> On Wed, Nov 18, 2020 at 5:12 AM Xintong Song <to...@gmail.com>
> wrote:
>
> > Hi devs,
> >
> > Updates on the progress of preparing the 1.11.3 release.
> >
> > We are approaching the creation of our first release candidate. Thanks
> all
> > for the efforts so far.
> >
> > # Ongoing threads
> >
> >    - *FLIP-27 backportings:* 74 out of 102 commits are already
> backported.
> >    Stephan and Becket are actively working on this.
> >    - *Blockers:* There are 2 remaining blockers, both have been fixed on
> >    the master branch and should be backported along with the FLIP-27
> > changes.
> >
> > As soon as the above threads are done, we will create our first release
> > candidate. If there's any other issues that you believe should be a
> release
> > blocker, please reach out to either me or Gordon.
> >
> > # JIRA version
> >
> > Version 1.11.4 has been created on JIRA. I've already tagged some of the
> > tickets with the new fix version. Please feel free to continue with the
> > efforts and merge once they are ready. I'll double check and update the
> fix
> > versions before the release.
> >
> > Thank you~
> >
> > Xintong Song
> >
> >
> >
> > On Thu, Nov 12, 2020 at 1:31 PM Xintong Song <to...@gmail.com>
> > wrote:
> >
> > > Thanks for the notice and fix, Roman.
> > >
> > > Thank you~
> > >
> > > Xintong Song
> > >
> > >
> > >
> > > On Wed, Nov 11, 2020 at 5:53 PM Khachatryan Roman <
> > > khachatryan.roman@gmail.com> wrote:
> > >
> > >> Hi,
> > >>
> > >> I'd like FLINK-20079 [1] to be merged into 1.11 and included in
> 1.11.3.
> > >>
> > >> [1] https://issues.apache.org/jira/browse/FLINK-20079
> > >>
> > >> Regards,
> > >> Roman
> > >>
> > >>
> > >> On Tue, Nov 10, 2020 at 8:21 AM Xintong Song <to...@gmail.com>
> > >> wrote:
> > >>
> > >> > Thanks for the notice, Dian.
> > >> >
> > >> > Thank you~
> > >> >
> > >> > Xintong Song
> > >> >
> > >> >
> > >> >
> > >> > On Tue, Nov 10, 2020 at 10:19 AM Dian Fu <di...@gmail.com>
> > wrote:
> > >> >
> > >> > > Hi Xintong,
> > >> > >
> > >> > > I want to bring one more issue to your attention [1]. The test
> case
> > >> > > UnalignedCheckpointCompatibilityITCase.test failed several times
> in
> > >> the
> > >> > > last nightly test of release-1.11. We need to figure out if it's
> > just
> > >> an
> > >> > > instable test or caused by recent changes.
> > >> > >
> > >> > > [1] https://issues.apache.org/jira/browse/FLINK-20065
> > >> > >
> > >> > > > 在 2020年11月10日，上午9:24，Xintong Song <to...@gmail.com> 写道：
> > >> > > >
> > >> > > > Thanks for the replies.
> > >> > > >
> > >> > > > Thank you~
> > >> > > >
> > >> > > > Xintong Song
> > >> > > >
> > >> > > >
> > >> > > >
> > >> > > > On Tue, Nov 10, 2020 at 1:09 AM Becket Qin <
> becket.qin@gmail.com>
> > >> > wrote:
> > >> > > >
> > >> > > >> Hi Xintong,
> > >> > > >>
> > >> > > >> Thanks for driving the release. Just want to sync up on the
> > FLIP-27
> > >> > > >> backporting. Stephan and I are still trying to backport a bunch
> > of
> > >> > > patches
> > >> > > >> of Source to 1.11.3. Including:
> > >> > > >>
> > >> > > >> [FLINK-19698][connector/common] Add a close() method to the
> > >> > SplitReader.
> > >> > > >> [FLINK-19717] SourceReaderBase.pollNext may return END_OF_INPUT
> > if
> > >> > > >> SplitReader.fetch throws
> > >> > > >> [FLINK-19535] [connector/common] Avoid failing a job multiple
> > >> times in
> > >> > > >> SourceCoordinator.
> > >> > > >> [FLINK-19265] [FLINK-20049][core] Source API final adjustments.
> > >> > > >>
> > >> > > >> and a few more fixes.
> > >> > > >>
> > >> > > >> We are currently trying to fix them in 1.12 first so it might
> > take
> > >> a
> > >> > > little
> > >> > > >> longer to backport them to 1.11.3. I think it will probably
> take
> > >> us a
> > >> > > few
> > >> > > >> more days to finish the backport. So that would roughly be the
> > end
> > >> of
> > >> > > this
> > >> > > >> week.
> > >> > > >>
> > >> > > >> Thanks,
> > >> > > >>
> > >> > > >> Jiangjie (Becket) Qin
> > >> > > >>
> > >> > > >>
> > >> > > >>
> > >> > > >>
> > >> > > >> On Mon, Nov 9, 2020 at 9:57 PM Till Rohrmann <
> > trohrmann@apache.org
> > >> >
> > >> > > wrote:
> > >> > > >>
> > >> > > >>> Yes, I've downgraded FLINK-19816 to critical.
> > >> > > >>>
> > >> > > >>> Cheers,
> > >> > > >>> Till
> > >> > > >>>
> > >> > > >>> On Mon, Nov 9, 2020 at 10:19 AM Xintong Song <
> > >> tonysong820@gmail.com>
> > >> > > >>> wrote:
> > >> > > >>>
> > >> > > >>>> Thanks for the notice, Till.
> > >> > > >>>>
> > >> > > >>>> I just checked and found FLINK-20033 is already fixed. Shall
> we
> > >> also
> > >> > > >>>> downgrade FLINK-19816 to `Critical`?
> > >> > > >>>>
> > >> > > >>>> Thank you~
> > >> > > >>>>
> > >> > > >>>> Xintong Song
> > >> > > >>>>
> > >> > > >>>>
> > >> > > >>>>
> > >> > > >>>> On Mon, Nov 9, 2020 at 4:42 PM Till Rohrmann <
> > >> trohrmann@apache.org>
> > >> > > >>> wrote:
> > >> > > >>>>
> > >> > > >>>>> I would like to bring one more critical issue to your
> > attention
> > >> > which
> > >> > > >>> is
> > >> > > >>>>> FLINK-20033 [1]. I believe that this issue is actually
> causing
> > >> what
> > >> > > >> has
> > >> > > >>>>> been reported in FLINK-19816 [2]. I hope to have it fixed by
> > the
> > >> > end
> > >> > > >> of
> > >> > > >>>>> today. Once FLINK-20033 is fixed, I think that we don't have
> > to
> > >> > block
> > >> > > >>> the
> > >> > > >>>>> release on FLINK-19816.
> > >> > > >>>>>
> > >> > > >>>>> [1] https://issues.apache.org/jira/browse/FLINK-20033
> > >> > > >>>>> [2] https://issues.apache.org/jira/browse/FLINK-19816
> > >> > > >>>>>
> > >> > > >>>>> Cheers,
> > >> > > >>>>> Till
> > >> > > >>>>>
> > >> > > >>>>> On Mon, Nov 9, 2020 at 4:05 AM Xintong Song <
> > >> tonysong820@gmail.com
> > >> > >
> > >> > > >>>> wrote:
> > >> > > >>>>>
> > >> > > >>>>>> Hi devs,
> > >> > > >>>>>>
> > >> > > >>>>>> I'd like to provide an update on the progress of preparing
> > >> release
> > >> > > >>>>> 1.11.3.
> > >> > > >>>>>>
> > >> > > >>>>>> *Blockers*
> > >> > > >>>>>> We currently have 3 remaining blockers. (3 resolved and 1
> > >> emerged
> > >> > > >>>>> compared
> > >> > > >>>>>> to last week)
> > >> > > >>>>>>
> > >> > > >>>>>> - [FLINK-19698] Add close() method and
> onCheckpointComplete()
> > >> to
> > >> > > >> the
> > >> > > >>>>>> Source.
> > >> > > >>>>>> The issue has been fixed on the master branch. It's
> currently
> > >> > > >> blocked
> > >> > > >>>> on
> > >> > > >>>>>> the FLIP-27 backportings to backport it to the 1.11 branch.
> > >> > > >>>>>>
> > >> > > >>>>>> - [FLINK-19717] SourceReaderBase.pollNext may return
> > >> END_OF_INPUT
> > >> > > >> if
> > >> > > >>>>>> SplitReader.fetch throws
> > >> > > >>>>>> A PR has been opened and reviewed. From the discussions on
> > the
> > >> PR,
> > >> > > >> it
> > >> > > >>>>> looks
> > >> > > >>>>>> close to mergeable.
> > >> > > >>>>>>
> > >> > > >>>>>> - [FLINK-19816] Flink restored from a wrong checkpoint (a
> > very
> > >> old
> > >> > > >>> one
> > >> > > >>>>> and
> > >> > > >>>>>> not the last completed one)
> > >> > > >>>>>> This is a newly emerged blocker and Matthias is working on
> > it.
> > >> > > >>>>>>
> > >> > > >>>>>> *Test Instabilities*
> > >> > > >>>>>> We currently have 27 test instabilities[1].
> > >> > > >>>>>> AFAIK, none of them are as serious as to block the 1.11.3
> > >> release.
> > >> > > >>>>>>
> > >> > > >>>>>> *FLIP-27 Backprotings*
> > >> > > >>>>>>
> > >> > > >>>>>> I noticed that there's no jira issues opened on the FLIP-27
> > >> > > >>> backporting
> > >> > > >>>>>> efforts, which is part of the major efforts planned for the
> > >> 1.11.3
> > >> > > >>>>> release,
> > >> > > >>>>>> making it hard to track the progress.
> > >> > > >>>>>>
> > >> > > >>>>>>
> > >> > > >>>>>> @Stephan and @Becket, could you please share the updates on
> > the
> > >> > > >>>>> backporting
> > >> > > >>>>>> efforts? How is the progress and when are the efforts
> > expected
> > >> to
> > >> > > >> be
> > >> > > >>>>>> finished? It would be appreciated and helpful if we can
> have
> > a
> > >> > jira
> > >> > > >>>>> ticket
> > >> > > >>>>>> to track the backporing progress publicly.
> > >> > > >>>>>>
> > >> > > >>>>>>
> > >> > > >>>>>> Thank you~
> > >> > > >>>>>>
> > >> > > >>>>>> Xintong Song
> > >> > > >>>>>>
> > >> > > >>>>>>
> > >> > > >>>>>> [1]
> > >> > > >>>>>>
> > >> > > >>>>>>
> > >> > > >>>>>
> > >> > > >>>>
> > >> > > >>>
> > >> > > >>
> > >> > >
> > >> >
> > >>
> >
> https://issues.apache.org/jira/browse/FLINK-16908?filter=12348580&jql=project%20%3D%20FLINK%20AND%20resolution%20%3D%20Unresolved%20AND%20affectedVersion%20in%20(1.11.0%2C%201.11.1%2C%201.11.2%2C%201.11.3)%20AND%20labels%20%3D%20test-stability%20ORDER%20BY%20updated%20DESC%2C%20created%20DESC
> > >> > > >>>>>>
> > >> > > >>>>>> On Wed, Nov 4, 2020 at 3:58 PM Tzu-Li (Gordon) Tai <
> > >> > > >>>> tzulitai@apache.org>
> > >> > > >>>>>> wrote:
> > >> > > >>>>>>
> > >> > > >>>>>>> Thanks for the efforts so far with pushing for 1.11.3.
> > >> > > >>>>>>>
> > >> > > >>>>>>> I'd like to provide a mid-week update on how we're looking
> > >> with
> > >> > > >> all
> > >> > > >>>> the
> > >> > > >>>>>>> pending blockers and backports:
> > >> > > >>>>>>>
> > >> > > >>>>>>> *Blockers**:*
> > >> > > >>>>>>>
> > >> > > >>>>>>> Currently all blockers either have an open PR, or have
> been
> > >> > > >> merged.
> > >> > > >>>>> I'll
> > >> > > >>>>>>> highlight below the blockers *that still require reviewing
> > >> > > >> efforts
> > >> > > >>>> for
> > >> > > >>>>>> them
> > >> > > >>>>>>> to move forward:*
> > >> > > >>>>>>>
> > >> > > >>>>>>> - [FLINK-19909] Flink application in attach mode could not
> > >> > > >>> terminate
> > >> > > >>>>> when
> > >> > > >>>>>>> the only job is canceled. PR:
> > >> > > >>>>> https://github.com/apache/flink/pull/13911
> > >> > > >>>>>>> - [FLINK-19717] SourceReaderBase.pollNext may return
> > >> END_OF_INPUT
> > >> > > >>> if
> > >> > > >>>>>>> SplitReader.fetch throws. PR:
> > >> > > >>>>> https://github.com/apache/flink/pull/13776
> > >> > > >>>>>>>
> > >> > > >>>>>>> The above PRs currently have no reviews at all yet. They
> > seem
> > >> to
> > >> > > >>>>> already
> > >> > > >>>>>>> have designated reviewers.
> > >> > > >>>>>>>
> > >> > > >>>>>>>
> > >> > > >>>>>>> *Backports:*
> > >> > > >>>>>>> - FLIP-27 robustness improvement backports: Stephan is
> > >> currently
> > >> > > >>>>> working
> > >> > > >>>>>> on
> > >> > > >>>>>>> backporting several FLIP-27 changes. There are no PRs yet
> > for
> > >> the
> > >> > > >>>>>>> backported changes.
> > >> > > >>>>>>> - DataStreamUtils.collect() refactorings backport: Steven
> Wu
> > >> > > >>>> mentioned
> > >> > > >>>>> to
> > >> > > >>>>>>> backport this, but AFAIK this isn't assigned to anyone
> yet.
> > >> > > >>>>>>>
> > >> > > >>>>>>> *ETAs*:
> > >> > > >>>>>>>
> > >> > > >>>>>>> I'd like to request ETAs for the remaining backports, to
> > >> prevent
> > >> > > >> a
> > >> > > >>>>> creep
> > >> > > >>>>>> in
> > >> > > >>>>>>> the scope of this bugfix release.
> > >> > > >>>>>>>
> > >> > > >>>>>>> We already have Flink users that would benefit from fixes
> > that
> > >> > > >> have
> > >> > > >>>>> been
> > >> > > >>>>>>> merged to release-1.11 already, so technically speaking
> the
> > >> > > >>> backports
> > >> > > >>>>>>> should be considered "nice-to-have" (to the best of my
> > >> knowledge
> > >> > > >> of
> > >> > > >>>> the
> > >> > > >>>>>>> changes) and could potentially be moved to a follow-up
> > 1.11.4.
> > >> > > >>>>>>> Most notably, the Stateful Functions project is already
> > >> waiting
> > >> > > >> on
> > >> > > >>>>> Flink
> > >> > > >>>>>>> 1.11.3 to address critical recovery issues (please see the
> > >> > > >> StateFun
> > >> > > >>>>> 2.2.1
> > >> > > >>>>>>> release discussion thread [1]).
> > >> > > >>>>>>>
> > >> > > >>>>>>> @Stephan Ewen <se...@apache.org> @Becket Qin <
> > >> > > >> becket.qin@gmail.com
> > >> > > >>>>
> > >> > > >>>>>> could
> > >> > > >>>>>>> you provide an ETA for the FLIP-27 backports? It would
> help
> > to
> > >> > > >> get
> > >> > > >>> a
> > >> > > >>>>>> better
> > >> > > >>>>>>> estimate to decide how we proceed here.
> > >> > > >>>>>>>
> > >> > > >>>>>>> Cheers,
> > >> > > >>>>>>> Gordon
> > >> > > >>>>>>>
> > >> > > >>>>>>> [1]
> > >> > > >>>>>>>
> > >> > > >>>>>>>
> > >> > > >>>>>>
> > >> > > >>>>>
> > >> > > >>>>
> > >> > > >>>
> > >> > > >>
> > >> > >
> > >> >
> > >>
> >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Releasing-StateFun-hotfix-version-2-2-1-td46239.html
> > >> > > >>>>>>>
> > >> > > >>>>>>> On Wed, Nov 4, 2020 at 3:16 PM Tzu-Li (Gordon) Tai <
> > >> > > >>>>> tzulitai@apache.org>
> > >> > > >>>>>>> wrote:
> > >> > > >>>>>>>
> > >> > > >>>>>>>>> The collect() utils can be picked back, I see no issue
> > with
> > >> > > >>> that
> > >> > > >>>>> (it
> > >> > > >>>>>> is
> > >> > > >>>>>>>> isolated utilities).
> > >> > > >>>>>>>>
> > >> > > >>>>>>>> Just checking on all the requested backports mentioned in
> > >> this
> > >> > > >>>>> thread,
> > >> > > >>>>>>> and
> > >> > > >>>>>>>> figuring out which ones seem to still be unassigned /
> open.
> > >> > > >>>>>>>>
> > >> > > >>>>>>>> Is someone working on backporting
> > >> > > >>>>>>>>
> > >> > > >>>>>>>
> > >> > > >>>>>>
> > >> > > >>>>>
> > >> > > >>>>
> > >> > > >>>
> > >> > > >>
> > >> > >
> > >> >
> > >>
> >
> https://github.com/apache/flink/commit/09a7a66b7313fea64817fe960a8da1265b428efc
> > >> > > >>>>>>>> to release-1.11 at the moment?
> > >> > > >>>>>>>>
> > >> > > >>>>>>>> On Wed, Nov 4, 2020 at 1:39 AM Steven Wu <
> > >> stevenz3wu@gmail.com
> > >> > > >>>
> > >> > > >>>>> wrote:
> > >> > > >>>>>>>>
> > >> > > >>>>>>>>> @Stephan Ewen <se...@apache.org> yeah, we can do that.
> > >> don't
> > >> > > >>>> worry
> > >> > > >>>>>>> about
> > >> > > >>>>>>>>> it. your earlier email had the perfect explanation on
> why
> > >> file
> > >> > > >>>>> source
> > >> > > >>>>>>>>> shouldn't be backported.
> > >> > > >>>>>>>>>
> > >> > > >>>>>>>>> On Tue, Nov 3, 2020 at 3:37 AM Stephan Ewen <
> > >> sewen@apache.org
> > >> > > >>>
> > >> > > >>>>> wrote:
> > >> > > >>>>>>>>>
> > >> > > >>>>>>>>>> @Steven would it be possible to initially copy some of
> > the
> > >> > > >>> code
> > >> > > >>>>> into
> > >> > > >>>>>>> the
> > >> > > >>>>>>>>>> iceberg source and later replace it by a dependency on
> > the
> > >> > > >>> Flink
> > >> > > >>>>>> file
> > >> > > >>>>>>>>>> source?
> > >> > > >>>>>>>>>>
> > >> > > >>>>>>>>>> On Mon, Nov 2, 2020 at 8:33 PM Steven Wu <
> > >> > > >>> stevenz3wu@gmail.com>
> > >> > > >>>>>>> wrote:
> > >> > > >>>>>>>>>>
> > >> > > >>>>>>>>>>> Stephan, thanks a lot for explaining the file
> connector.
> > >> > > >>> that
> > >> > > >>>>>> makes
> > >> > > >>>>>>>>>> sense.
> > >> > > >>>>>>>>>>>
> > >> > > >>>>>>>>>>> I was asking because we were trying to reuse some of
> the
> > >> > > >>>>>>>>> implementations
> > >> > > >>>>>>>>>> in
> > >> > > >>>>>>>>>>> the file source for Iceberg source. Flink Iceberg
> source
> > >> > > >>> lives
> > >> > > >>>>> in
> > >> > > >>>>>>> the
> > >> > > >>>>>>>>>>> Iceberg repo, which is not possible to code against
> the
> > >> > > >>> master
> > >> > > >>>>>>> branch
> > >> > > >>>>>>>>> of
> > >> > > >>>>>>>>>>> the Flink code.
> > >> > > >>>>>>>>>>>
> > >> > > >>>>>>>>>>> On Mon, Nov 2, 2020 at 3:31 AM Stephan Ewen <
> > >> > > >>> sewen@apache.org
> > >> > > >>>>>
> > >> > > >>>>>>> wrote:
> > >> > > >>>>>>>>>>>
> > >> > > >>>>>>>>>>>> Hi Steven!
> > >> > > >>>>>>>>>>>>
> > >> > > >>>>>>>>>>>> So far there are no plans to pick back the file
> system
> > >> > > >>>>> connector
> > >> > > >>>>>>>>> code.
> > >> > > >>>>>>>>>>> This
> > >> > > >>>>>>>>>>>> is still evolving and not finalized for 1.12, so I
> > don't
> > >> > > >>>> feel
> > >> > > >>>>> it
> > >> > > >>>>>>> is
> > >> > > >>>>>>>>> a
> > >> > > >>>>>>>>>>> good
> > >> > > >>>>>>>>>>>> candidate to be backported.
> > >> > > >>>>>>>>>>>> However, with the base connector changes backported,
> > you
> > >> > > >>>>> should
> > >> > > >>>>>> be
> > >> > > >>>>>>>>> able
> > >> > > >>>>>>>>>>> to
> > >> > > >>>>>>>>>>>> run the file connector code from master against
> 1.11.3.
> > >> > > >>>>>>>>>>>>
> > >> > > >>>>>>>>>>>> The collect() utils can be picked back, I see no
> issue
> > >> > > >>> with
> > >> > > >>>>> that
> > >> > > >>>>>>>>> (it is
> > >> > > >>>>>>>>>>>> isolated utilities).
> > >> > > >>>>>>>>>>>>
> > >> > > >>>>>>>>>>>> Best,
> > >> > > >>>>>>>>>>>> Stephan
> > >> > > >>>>>>>>>>>>
> > >> > > >>>>>>>>>>>>
> > >> > > >>>>>>>>>>>> On Mon, Nov 2, 2020 at 3:02 AM Steven Wu <
> > >> > > >>>>> stevenz3wu@gmail.com>
> > >> > > >>>>>>>>> wrote:
> > >> > > >>>>>>>>>>>>
> > >> > > >>>>>>>>>>>>> Basically, it would be great to get the latest code
> in
> > >> > > >>> the
> > >> > > >>>>>>>>>>>>> flink-connector-files (FLIP-27).
> > >> > > >>>>>>>>>>>>>
> > >> > > >>>>>>>>>>>>> On Sat, Oct 31, 2020 at 9:57 AM Steven Wu <
> > >> > > >>>>>> stevenz3wu@gmail.com
> > >> > > >>>>>>>>
> > >> > > >>>>>>>>>>> wrote:
> > >> > > >>>>>>>>>>>>>
> > >> > > >>>>>>>>>>>>>> Stephan, it will be great if we can also backport
> > >> > > >> the
> > >> > > >>>>>>>>>> DataStreamUtils
> > >> > > >>>>>>>>>>>>>> related commits that help with collecting output
> > >> > > >> from
> > >> > > >>>>>>> unbounded
> > >> > > >>>>>>>>>>>> streams.
> > >> > > >>>>>>>>>>>>>> e.g.
> > >> > > >>>>>>>>>>>>>>
> > >> > > >>>>>>>>>>>>>>
> > >> > > >>>>>>>>>>>>>>
> > >> > > >>>>>>>>>>>>>
> > >> > > >>>>>>>>>>>>
> > >> > > >>>>>>>>>>>
> > >> > > >>>>>>>>>>
> > >> > > >>>>>>>>>
> > >> > > >>>>>>>
> > >> > > >>>>>>
> > >> > > >>>>>
> > >> > > >>>>
> > >> > > >>>
> > >> > > >>
> > >> > >
> > >> >
> > >>
> >
> https://github.com/apache/flink/commit/09a7a66b7313fea64817fe960a8da1265b428efc
> > >> > > >>>>>>>>>>>>>>
> > >> > > >>>>>>>>>>>>>> I tried to copy and paste the code to unblock
> > >> > > >> myself.
> > >> > > >>>> but
> > >> > > >>>>> it
> > >> > > >>>>>>>>>> quickly
> > >> > > >>>>>>>>>>>> got
> > >> > > >>>>>>>>>>>>>> into the rabbit hole of more and more code.
> > >> > > >>>>>>>>>>>>>>
> > >> > > >>>>>>>>>>>>>> On Fri, Oct 30, 2020 at 11:02 AM Stephan Ewen <
> > >> > > >>>>>>> sewen@apache.org
> > >> > > >>>>>>>>>>
> > >> > > >>>>>>>>>>>> wrote:
> > >> > > >>>>>>>>>>>>>>
> > >> > > >>>>>>>>>>>>>>> I have started with backporting the source API
> > >> > > >>> changes.
> > >> > > >>>>>> Some
> > >> > > >>>>>>>>> minor
> > >> > > >>>>>>>>>>>>>>> conflicts to solve, will need a bit more to finish
> > >> > > >>>> this.
> > >> > > >>>>>>>>>>>>>>>
> > >> > > >>>>>>>>>>>>>>> On Fri, Oct 30, 2020 at 7:25 AM Tzu-Li (Gordon)
> > >> > > >> Tai <
> > >> > > >>>>>>>>>>>>> tzulitai@apache.org>
> > >> > > >>>>>>>>>>>>>>> wrote:
> > >> > > >>>>>>>>>>>>>>>
> > >> > > >>>>>>>>>>>>>>>> @Stephan Ewen <se...@apache.org>
> > >> > > >>>>>>>>>>>>>>>> Are there already plans or ongoing efforts for
> > >> > > >>>>>> backporting
> > >> > > >>>>>>>>> the
> > >> > > >>>>>>>>>>> list
> > >> > > >>>>>>>>>>>> of
> > >> > > >>>>>>>>>>>>>>>> FLIP-27 changes that you posted?
> > >> > > >>>>>>>>>>>>>>>>
> > >> > > >>>>>>>>>>>>>>>> On Thu, Oct 29, 2020 at 7:08 PM Xintong Song <
> > >> > > >>>>>>>>>>> tonysong820@gmail.com
> > >> > > >>>>>>>>>>>>>
> > >> > > >>>>>>>>>>>>>>>> wrote:
> > >> > > >>>>>>>>>>>>>>>>
> > >> > > >>>>>>>>>>>>>>>>> Hi folks,
> > >> > > >>>>>>>>>>>>>>>>>
> > >> > > >>>>>>>>>>>>>>>>> Just to provide some updates concerning the
> > >> > > >> status
> > >> > > >>>> on
> > >> > > >>>>>> the
> > >> > > >>>>>>>>>>>>>>>>> test instabilities.
> > >> > > >>>>>>>>>>>>>>>>>
> > >> > > >>>>>>>>>>>>>>>>> Currently, we have 30 unresolved tickets labeled
> > >> > > >>>> with
> > >> > > >>>>>>>>> `Affects
> > >> > > >>>>>>>>>>>>> Version`
> > >> > > >>>>>>>>>>>>>>>>> 1.11.x.
> > >> > > >>>>>>>>>>>>>>>>>
> > >> > > >>>>>>>>>>>>>>>>>
> > >> > > >>>>>>>>>>>>>>>
> > >> > > >>>>>>>>>>>>>
> > >> > > >>>>>>>>>>>>
> > >> > > >>>>>>>>>>>
> > >> > > >>>>>>>>>>
> > >> > > >>>>>>>>>
> > >> > > >>>>>>>
> > >> > > >>>>>>
> > >> > > >>>>>
> > >> > > >>>>
> > >> > > >>>
> > >> > > >>
> > >> > >
> > >> >
> > >>
> >
> https://issues.apache.org/jira/browse/FLINK-19775?filter=12348580&jql=project%20%3D%20FLINK%20AND%20resolution%20%3D%20Unresolved%20AND%20affectedVersion%20in%20(1.11.0%2C%201.11.1%2C%201.11.2%2C%201.11.3)%20AND%20labels%20%3D%20test-stability%20ORDER%20BY%20created%20DESC
> > >> > > >>>>>>>>>>>>>>>>>
> > >> > > >>>>>>>>>>>>>>>>> Among the 30 tickets, 11 of them are:
> > >> > > >>>>>>>>>>>>>>>>> - Have occured in the recent 3 months
> > >> > > >>>>>>>>>>>>>>>>> - Not confirmed to be pure testability issues
> > >> > > >>>>>>>>>>>>>>>>> - Not confirmed to be rare condition cases
> > >> > > >>>>>>>>>>>>>>>>>
> > >> > > >>>>>>>>>>>>>>>>> It would be nice if someone familiar with these
> > >> > > >>>>>> components
> > >> > > >>>>>>>>> can
> > >> > > >>>>>>>>>>>> take a
> > >> > > >>>>>>>>>>>>>>> look
> > >> > > >>>>>>>>>>>>>>>>> into these issues.
> > >> > > >>>>>>>>>>>>>>>>>
> > >> > > >>>>>>>>>>>>>>>>> -
> > >> > > >>> https://issues.apache.org/jira/browse/FLINK-17159
> > >> > > >>>>>> (ES6)
> > >> > > >>>>>>>>>>>>>>>>> -
> > >> > > >>> https://issues.apache.org/jira/browse/FLINK-17912
> > >> > > >>>>>>> (Kafka)
> > >> > > >>>>>>>>>>>>>>>>> -
> > >> > > >>> https://issues.apache.org/jira/browse/FLINK-17949
> > >> > > >>>>>>> (Kafka)
> > >> > > >>>>>>>>>>>>>>>>> ⁃
> > >> > > >>> https://issues.apache.org/jira/browse/FLINK-18444
> > >> > > >>>>>>> (Kafka)
> > >> > > >>>>>>>>>>>>>>>>> -
> > >> > > >>> https://issues.apache.org/jira/browse/FLINK-18634
> > >> > > >>>>>>> (Kafka)
> > >> > > >>>>>>>>>>>>>>>>> -
> > >> > > >>> https://issues.apache.org/jira/browse/FLINK-18648
> > >> > > >>>>>>> (Kafka)
> > >> > > >>>>>>>>>>>>>>>>> -
> > >> > > >>> https://issues.apache.org/jira/browse/FLINK-18807
> > >> > > >>>>>>> (Kafka)
> > >> > > >>>>>>>>>>>>>>>>> -
> > >> > > >>> https://issues.apache.org/jira/browse/FLINK-19369
> > >> > > >>>>>>>>>>>> (BlobClientTest)
> > >> > > >>>>>>>>>>>>>>>>> -
> > >> > > >>> https://issues.apache.org/jira/browse/FLINK-19436
> > >> > > >>>>>>> (TPCDS)
> > >> > > >>>>>>>>>>>>>>>>> -
> > >> > > >>> https://issues.apache.org/jira/browse/FLINK-19690
> > >> > > >>>>>>>>>>>> (Format/Parquet)
> > >> > > >>>>>>>>>>>>>>>>> -
> > >> > > >>> https://issues.apache.org/jira/browse/FLINK-19775
> > >> > > >>>>>>>>>>>>>>>>> (SystemProcessingTimeServiceTest)
> > >> > > >>>>>>>>>>>>>>>>>
> > >> > > >>>>>>>>>>>>>>>>> Thank you~
> > >> > > >>>>>>>>>>>>>>>>>
> > >> > > >>>>>>>>>>>>>>>>> Xintong Song
> > >> > > >>>>>>>>>>>>>>>>>
> > >> > > >>>>>>>>>>>>>>>>>
> > >> > > >>>>>>>>>>>>>>>>>
> > >> > > >>>>>>>>>>>>>>>>> On Thu, Oct 29, 2020 at 10:21 AM Jingsong Li <
> > >> > > >>>>>>>>>>>> jingsonglee0@gmail.com
> > >> > > >>>>>>>>>>>>>>
> > >> > > >>>>>>>>>>>>>>>>> wrote:
> > >> > > >>>>>>>>>>>>>>>>>
> > >> > > >>>>>>>>>>>>>>>>>> +1 to backport the FLIP-27 adjustments to
> > >> > > >>> 1.11.x.
> > >> > > >>>>>>>>>>>>>>>>>>
> > >> > > >>>>>>>>>>>>>>>>>> If possible, that would be great. Many people
> > >> > > >>> are
> > >> > > >>>>>>> looking
> > >> > > >>>>>>>>>>> forward
> > >> > > >>>>>>>>>>>>> to
> > >> > > >>>>>>>>>>>>>>> the
> > >> > > >>>>>>>>>>>>>>>>>> FLIP-27 interface, but they don't want to take
> > >> > > >>> the
> > >> > > >>>>>> risk
> > >> > > >>>>>>> to
> > >> > > >>>>>>>>>>>> upgrade
> > >> > > >>>>>>>>>>>>> to
> > >> > > >>>>>>>>>>>>>>>>> 1.12
> > >> > > >>>>>>>>>>>>>>>>>> (And wait 1.12). After all, 1.11 is a
> > >> > > >> relatively
> > >> > > >>>>>> stable
> > >> > > >>>>>>>>>>> version.
> > >> > > >>>>>>>>>>>>>>>>>>
> > >> > > >>>>>>>>>>>>>>>>>> Best,
> > >> > > >>>>>>>>>>>>>>>>>> Jingsong
> > >> > > >>>>>>>>>>>>>>>>>>
> > >> > > >>>>>>>>>>>>>>>>>> On Thu, Oct 29, 2020 at 1:24 AM Stephan Ewen <
> > >> > > >>>>>>>>>> sewen@apache.org
> > >> > > >>>>>>>>>>>>
> > >> > > >>>>>>>>>>>>>>> wrote:
> > >> > > >>>>>>>>>>>>>>>>>>
> > >> > > >>>>>>>>>>>>>>>>>>> Thanks for starting this.
> > >> > > >>>>>>>>>>>>>>>>>>>
> > >> > > >>>>>>>>>>>>>>>>>>> +1 form my side to backport the FLIP-27
> > >> > > >>>>> adjustments
> > >> > > >>>>>> to
> > >> > > >>>>>>>>>>> 1.11.x.
> > >> > > >>>>>>>>>>>>>>>>>>>
> > >> > > >>>>>>>>>>>>>>>>>>> There were quite a few changes, and I think
> > >> > > >> we
> > >> > > >>>>> need
> > >> > > >>>>>> to
> > >> > > >>>>>>>>>>>>> cherry-pick
> > >> > > >>>>>>>>>>>>>>>>> them
> > >> > > >>>>>>>>>>>>>>>>>> all
> > >> > > >>>>>>>>>>>>>>>>>>> to not get some inconsistent mix of changes
> > >> > > >>> and
> > >> > > >>>>> many
> > >> > > >>>>>>>>> merge
> > >> > > >>>>>>>>>>>>>>> conflicts.
> > >> > > >>>>>>>>>>>>>>>>>>> I made a list below of what we need to add
> > >> > > >> to
> > >> > > >>>>>>>>>> "release-1.11".
> > >> > > >>>>>>>>>>>>>>>>>>>
> > >> > > >>>>>>>>>>>>>>>>>>> *#### Core Source API Changes to backport
> > >> > > >> (in
> > >> > > >>>>>> REVERSE
> > >> > > >>>>>>>>>> order)*
> > >> > > >>>>>>>>>>>>>>>>>>>
> > >> > > >>>>>>>>>>>>>>>>>>>  (Use: "git log
> > >> > > >>>>>>>>>>>>>>>>>>>
> > >> > > >>>>>>>>>>>>
> > >> > > >>>>>
> > flink-core/src/main/java/org/apache/flink/api/connector/source")
> > >> > > >>>>>>>>>>>>>>>>>>>
> > >> > > >>>>>>>>>>>>>>>>>>> commit
> > >> > > >>> 162c072e9265a7b6dd9d6f5459eb7974091c4c4e
> > >> > > >>>>>>>>>>>>>>>>>>> [FLINK-19492][core] Consolidate Source
> > >> > > >> Events
> > >> > > >>>>>> between
> > >> > > >>>>>>>>>> Source
> > >> > > >>>>>>>>>>>> API
> > >> > > >>>>>>>>>>>>>>> and
> > >> > > >>>>>>>>>>>>>>>>>> Split
> > >> > > >>>>>>>>>>>>>>>>>>> Reader API
> > >> > > >>>>>>>>>>>>>>>>>>>
> > >> > > >>>>>>>>>>>>>>>>>>> commit
> > >> > > >>> 428fbbad85379f71a9fbf1a1ab3e3dcfa9006436
> > >> > > >>>>>>>>>>>>>>>>>>> [FLINK-19205][core] Add access to
> > >> > > >>> configuration
> > >> > > >>>>> and
> > >> > > >>>>>>>>>> hostname
> > >> > > >>>>>>>>>>> in
> > >> > > >>>>>>>>>>>>> the
> > >> > > >>>>>>>>>>>>>>>>>>> SourceReaderContext
> > >> > > >>>>>>>>>>>>>>>>>>>
> > >> > > >>>>>>>>>>>>>>>>>>> commit
> > >> > > >>> 723e1790744ddc78d19c7c978442af1383f38d33
> > >> > > >>>>>>>>>>>>>>>>>>> [hotfix][core] Add to Source Enumerator
> > >> > > >>>>> convenience
> > >> > > >>>>>>>>> methods
> > >> > > >>>>>>>>>>> to
> > >> > > >>>>>>>>>>>>>>> assign
> > >> > > >>>>>>>>>>>>>>>>>>> single split
> > >> > > >>>>>>>>>>>>>>>>>>>
> > >> > > >>>>>>>>>>>>>>>>>>> *#### Connector Base Changes to Backport (in
> > >> > > >>>>> REVERSE
> > >> > > >>>>>>>>>> order)*
> > >> > > >>>>>>>>>>>>>>>>>>>
> > >> > > >>>>>>>>>>>>>>>>>>>  (Use: "git log
> > >> > > >>>>>>> flink-connectors/flink-connector-base")
> > >> > > >>>>>>>>>>>>>>>>>>>
> > >> > > >>>>>>>>>>>>>>>>>>> commit
> > >> > > >>> 401f56fe9d6b0271260edf9787cdcbfe4d03874d
> > >> > > >>>>>>>>>>>>>>>>>>> [FLINK-19427][FLINK-19489][tests] Fix test
> > >> > > >>>>>> conditions
> > >> > > >>>>>>>>> for
> > >> > > >>>>>>>>>>>>>>>>>>>
> > >> > > >>>>>>> 'SplitFetcherTest.testNotifiesWhenGoingIdleConcurrent()'
> > >> > > >>>>>>>>>>>>>>>>>>>
> > >> > > >>>>>>>>>>>>>>>>>>> commit
> > >> > > >>> 68c5c2ff779d82a1ff81ffaf60d8a1b283797db1
> > >> > > >>>>>>>>>>>>>>>>>>> [FLINK-19448][connector base] Explicitly
> > >> > > >> check
> > >> > > >>>> for
> > >> > > >>>>>>>>>>> un-expected
> > >> > > >>>>>>>>>>>>>>>>> condition
> > >> > > >>>>>>>>>>>>>>>>>>> that would leave an inconsistent state
> > >> > > >>>>>>>>>>>>>>>>>>>
> > >> > > >>>>>>>>>>>>>>>>>>> commit
> > >> > > >>> 162c072e9265a7b6dd9d6f5459eb7974091c4c4e
> > >> > > >>>>>>>>>>>>>>>>>>> [FLINK-19492][core] Consolidate Source
> > >> > > >> Events
> > >> > > >>>>>> between
> > >> > > >>>>>>>>>> Source
> > >> > > >>>>>>>>>>>> API
> > >> > > >>>>>>>>>>>>>>> and
> > >> > > >>>>>>>>>>>>>>>>>> Split
> > >> > > >>>>>>>>>>>>>>>>>>> Reader API
> > >> > > >>>>>>>>>>>>>>>>>>>
> > >> > > >>>>>>>>>>>>>>>>>>> commit
> > >> > > >>> c1ca7a4c7c21ec8868c14cdddd43c559625b794c
> > >> > > >>>>>>>>>>>>>>>>>>> [refactor][tests] Move some source test
> > >> > > >> utils
> > >> > > >>>> from
> > >> > > >>>>>>>>>>>>>>>>> flink-connector-base
> > >> > > >>>>>>>>>>>>>>>>>> to
> > >> > > >>>>>>>>>>>>>>>>>>> flink-core
> > >> > > >>>>>>>>>>>>>>>>>>>
> > >> > > >>>>>>>>>>>>>>>>>>> commit
> > >> > > >>> ee5c4c211c35c70d28252363bbc8400453609977
> > >> > > >>>>>>>>>>>>>>>>>>> [FLINK-19251][connectors] Avoid confusing
> > >> > > >>> queue
> > >> > > >>>>>>>>> handling in
> > >> > > >>>>>>>>>>>>>>>>>>> "SplitReader.handleSplitsChanges()"
> > >> > > >>>>>>>>>>>>>>>>>>>
> > >> > > >>>>>>>>>>>>>>>>>>> commit
> > >> > > >>> 5abef56b2bf85bcac786f6b16b6899b6cced7176
> > >> > > >>>>>>>>>>>>>>>>>>> [FLINK-19250][connectors] Fix error
> > >> > > >>> propagation
> > >> > > >>>> in
> > >> > > >>>>>>>>>> connector
> > >> > > >>>>>>>>>>>> base
> > >> > > >>>>>>>>>>>>>>>>>>> (SplitFetcherManager).
> > >> > > >>>>>>>>>>>>>>>>>>>
> > >> > > >>>>>>>>>>>>>>>>>>> commit
> > >> > > >>> 8fcca837c55a9216595ee4c03038b52747098dbb
> > >> > > >>>>>>>>>>>>>>>>>>> [hotfix][connectors] Improve JavaDocs for
> > >> > > >>>>>>>>>>>>>>> SingleThreadFetcherManager
> > >> > > >>>>>>>>>>>>>>>>>>>
> > >> > > >>>>>>>>>>>>>>>>>>> commit
> > >> > > >>> 4700bb5dde3303cbe98882f6beb7379425717b01
> > >> > > >>>>>>>>>>>>>>>>>>> [FLINK-19225][connectors] Various small
> > >> > > >>>>> improvements
> > >> > > >>>>>>> to
> > >> > > >>>>>>>>>>>>>>>>> SourceReaderBase
> > >> > > >>>>>>>>>>>>>>>>>>> (part 2)
> > >> > > >>>>>>>>>>>>>>>>>>>
> > >> > > >>>>>>>>>>>>>>>>>>> commit
> > >> > > >>> 12261c6b7ed6478a9b9f6a69cb58246b83cab9b7
> > >> > > >>>>>>>>>>>>>>>>>>> [FLINK-17393][connectors] (follow-up) Wakeup
> > >> > > >>> the
> > >> > > >>>>>>>>>>> SplitFetchers
> > >> > > >>>>>>>>>>>>> more
> > >> > > >>>>>>>>>>>>>>>>>>> elegantly.
> > >> > > >>>>>>>>>>>>>>>>>>>
> > >> > > >>>>>>>>>>>>>>>>>>> commit
> > >> > > >>> c60aaff0249bfd6b5871b7f82e03efc487a54d6b
> > >> > > >>>>>>>>>>>>>>>>>>> [hotfix][tests] Extend test coverage for
> > >> > > >>>>>>>>>>>>>>> FutureCompletingBlockingQueue
> > >> > > >>>>>>>>>>>>>>>>>>>
> > >> > > >>>>>>>>>>>>>>>>>>> commit
> > >> > > >>> cef8a587d7fd2fe64cc644da5ed095d82e46f631
> > >> > > >>>>>>>>>>>>>>>>>>> [FLINK-19245][connectors] Set default
> > >> > > >> capacity
> > >> > > >>>> for
> > >> > > >>>>>>>>>>>>>>>>>>> FutureCompletingBlockingQueue.
> > >> > > >>>>>>>>>>>>>>>>>>>
> > >> > > >>>>>>>>>>>>>>>>>>> commit
> > >> > > >>> 4ea95782b4c6a2538153d4d16ad3f4839c7de0fb
> > >> > > >>>>>>>>>>>>>>>>>>> [FLINK-19223][connectors] Simplify
> > >> > > >>> Availability
> > >> > > >>>>>> Future
> > >> > > >>>>>>>>>> Model
> > >> > > >>>>>>>>>>> in
> > >> > > >>>>>>>>>>>>>>> Base
> > >> > > >>>>>>>>>>>>>>>>>>> Connector
> > >> > > >>>>>>>>>>>>>>>>>>>
> > >> > > >>>>>>>>>>>>>>>>>>> commit
> > >> > > >>> 511857049ba30c8ff0ee56da551fa4a479dc583e
> > >> > > >>>>>>>>>>>>>>>>>>> [FLINK-18128][connectors] Ensure idle split
> > >> > > >>>>> fetchers
> > >> > > >>>>>>>>> lead
> > >> > > >>>>>>>>>> to
> > >> > > >>>>>>>>>>>>>>>>> availability
> > >> > > >>>>>>>>>>>>>>>>>>> notifications.
> > >> > > >>>>>>>>>>>>>>>>>>>
> > >> > > >>>>>>>>>>>>>>>>>>> commit
> > >> > > >>> a8206467af0830dcb89623ea068b5ca3b3450c92
> > >> > > >>>>>>>>>>>>>>>>>>> [refactor][core] Eagerly initialize the
> > >> > > >>>> FetchTask
> > >> > > >>>>> to
> > >> > > >>>>>>>>>> support
> > >> > > >>>>>>>>>>>>> proper
> > >> > > >>>>>>>>>>>>>>>>> unit
> > >> > > >>>>>>>>>>>>>>>>>>> testing
> > >> > > >>>>>>>>>>>>>>>>>>>
> > >> > > >>>>>>>>>>>>>>>>>>> commit
> > >> > > >>> 3b2f54bcb437f98e6137c904045cc51072b5c06b
> > >> > > >>>>>>>>>>>>>>>>>>> [hotfix][tests] Move constants in
> > >> > > >>>> SplitFetcherTest
> > >> > > >>>>>>>>> relevant
> > >> > > >>>>>>>>>>> to
> > >> > > >>>>>>>>>>>>> only
> > >> > > >>>>>>>>>>>>>>>>> one
> > >> > > >>>>>>>>>>>>>>>>>>> test into test method
> > >> > > >>>>>>>>>>>>>>>>>>>
> > >> > > >>>>>>>>>>>>>>>>>>> commit
> > >> > > >>> d7625760a75a508bf05bcddc380bb4d62ee1743e
> > >> > > >>>>>>>>>>>>>>>>>>> [FLINK-19225][connectors] Various small
> > >> > > >>>>> improvements
> > >> > > >>>>>>> to
> > >> > > >>>>>>>>>>>>>>>>> SourceReaderBase
> > >> > > >>>>>>>>>>>>>>>>>>>
> > >> > > >>>>>>>>>>>>>>>>>>> commit
> > >> > > >>> a5b0d3297748c1be47ad579a88f24df2255a8df1
> > >> > > >>>>>>>>>>>>>>>>>>> [FLINK-17393][connectors] Wakeup the
> > >> > > >>>> SplitFetchers
> > >> > > >>>>>>> more
> > >> > > >>>>>>>>>>>>> elegantly.
> > >> > > >>>>>>>>>>>>>>>>>>>
> > >> > > >>>>>>>>>>>>>>>>>>> commit
> > >> > > >>> f42a3ebc3e81a034b7221a803c153636fef34903
> > >> > > >>>>>>>>>>>>>>>>>>> [FLINK-18680][connectors] Make connector
> > >> > > >> base
> > >> > > >>>>>>>>>>>> RecordsWithSplitIds
> > >> > > >>>>>>>>>>>>>>> more
> > >> > > >>>>>>>>>>>>>>>>>>> lightweight.
> > >> > > >>>>>>>>>>>>>>>>>>>
> > >> > > >>>>>>>>>>>>>>>>>>> commit
> > >> > > >>> e3d273de822b085183d09b275a445879ff94b350
> > >> > > >>>>>>>>>>>>>>>>>>> [FLINK-19162][connectors] Add 'recycle()' to
> > >> > > >>> the
> > >> > > >>>>>>>>>>>>>>> RecordsWithSplitIds
> > >> > > >>>>>>>>>>>>>>>>> to
> > >> > > >>>>>>>>>>>>>>>>>>> support reuse of heavy objects.
> > >> > > >>>>>>>>>>>>>>>>>>>
> > >> > > >>>>>>>>>>>>>>>>>>> commit
> > >> > > >>> 8ebc464c2520453a70001cd712abc8dee6ee89e0
> > >> > > >>>>>>>>>>>>>>>>>>> [hotfix][testing] Add a set of
> > >> > > >> parameterizable
> > >> > > >>>>>> testing
> > >> > > >>>>>>>>>> mocks
> > >> > > >>>>>>>>>>>> for
> > >> > > >>>>>>>>>>>>>>> the
> > >> > > >>>>>>>>>>>>>>>>>> Split
> > >> > > >>>>>>>>>>>>>>>>>>> Reader API
> > >> > > >>>>>>>>>>>>>>>>>>>
> > >> > > >>>>>>>>>>>>>>>>>>> commit
> > >> > > >>> 930a07438be1185388d7150640f294dfe2a5d378
> > >> > > >>>>>>>>>>>>>>>>>>> [hotfix][connectors] Add RequestSplitEvent
> > >> > > >> to
> > >> > > >>>>>>>>>>>>>>> 'flink-connector-base'
> > >> > > >>>>>>>>>>>>>>>>>>>
> > >> > > >>>>>>>>>>>>>>>>>>> commit
> > >> > > >>> e945ce8a933bc378844782f784ca473c767ca159
> > >> > > >>>>>>>>>>>>>>>>>>> [refactor][connectors] Backport of the
> > >> > > >>>>>> connector-base
> > >> > > >>>>>>>>>>> exception
> > >> > > >>>>>>>>>>>>>>>>> handling
> > >> > > >>>>>>>>>>>>>>>>>>> from the Kafka Connector Pull Request
> > >> > > >>>>>>>>>>>>>>>>>>>
> > >> > > >>>>>>>>>>>>>>>>>>>
> > >> > > >>>>>>>>>>>>>>>>>>> On Wed, Oct 28, 2020 at 5:49 PM Steven Wu <
> > >> > > >>>>>>>>>>>> stevenz3wu@gmail.com>
> > >> > > >>>>>>>>>>>>>>>>> wrote:
> > >> > > >>>>>>>>>>>>>>>>>>>
> > >> > > >>>>>>>>>>>>>>>>>>>> I would love to see this FLIP-27 source
> > >> > > >>>>> interface
> > >> > > >>>>>>>>>>> improvement
> > >> > > >>>>>>>>>>>>> [1]
> > >> > > >>>>>>>>>>>>>>>>> made
> > >> > > >>>>>>>>>>>>>>>>>> to
> > >> > > >>>>>>>>>>>>>>>>>>>> 1.11.3.
> > >> > > >>>>>>>>>>>>>>>>>>>>
> > >> > > >>>>>>>>>>>>>>>>>>>> [1]
> > >> > > >>>>>>> https://issues.apache.org/jira/browse/FLINK-19698
> > >> > > >>>>>>>>>>>>>>>>>>>>
> > >> > > >>>>>>>>>>>>>>>>>>>> On Wed, Oct 28, 2020 at 12:32 AM Tzu-Li
> > >> > > >>>> (Gordon)
> > >> > > >>>>>>> Tai <
> > >> > > >>>>>>>>>>>>>>>>>>> tzulitai@apache.org>
> > >> > > >>>>>>>>>>>>>>>>>>>> wrote:
> > >> > > >>>>>>>>>>>>>>>>>>>>
> > >> > > >>>>>>>>>>>>>>>>>>>>> Thanks for the replies so far!
> > >> > > >>>>>>>>>>>>>>>>>>>>>
> > >> > > >>>>>>>>>>>>>>>>>>>>> Just to provide a brief update on the
> > >> > > >>> status
> > >> > > >>>>> of
> > >> > > >>>>>>>>>> blockers
> > >> > > >>>>>>>>>>>> for
> > >> > > >>>>>>>>>>>>>>>>> 1.11.3
> > >> > > >>>>>>>>>>>>>>>>>> so
> > >> > > >>>>>>>>>>>>>>>>>>>> far:
> > >> > > >>>>>>>>>>>>>>>>>>>>>
> > >> > > >>>>>>>>>>>>>>>>>>>>>
> > >> > > >>>>>>>>>>>>>>>>>>>>> *PR opened, pending reviewer*-
> > >> > > >>> [FLINK-19717]
> > >> > > >>>>>>>>>>>>>>>>>> SourceReaderBase.pollNext
> > >> > > >>>>>>>>>>>>>>>>>>>> may
> > >> > > >>>>>>>>>>>>>>>>>>>>> return END_OF_INPUT if SplitReader.fetch
> > >> > > >>>>> throws
> > >> > > >>>>>> (
> > >> > > >>>>>>>>>>>>>>>>>>>>>
> > >> > > >>> https://github.com/apache/flink/pull/13776)
> > >> > > >>>>>>>>>>>>>>>>>>>>>
> > >> > > >>>>>>>>>>>>>>>>>>>>> *PR opened, reviewed + close to being
> > >> > > >>>> merged*
> > >> > > >>>>>>>>>>>>>>>>>>>>> - [FLINK-19741] Timer service should
> > >> > > >> skip
> > >> > > >>>>>>> restoring
> > >> > > >>>>>>>>>> from
> > >> > > >>>>>>>>>>>> raw
> > >> > > >>>>>>>>>>>>>>> keyed
> > >> > > >>>>>>>>>>>>>>>>>>> stream
> > >> > > >>>>>>>>>>>>>>>>>>>>> if it isn't the writer (
> > >> > > >>>>>>>>>>>>>>>>> https://github.com/apache/flink/pull/13761)
> > >> > > >>>>>>>>>>>>>>>>>>>>> - [FLINK-19748] Raw keyed stream key
> > >> > > >> group
> > >> > > >>>>>>> iterator
> > >> > > >>>>>>>>>>> should
> > >> > > >>>>>>>>>>>> be
> > >> > > >>>>>>>>>>>>>>>>>> skipping
> > >> > > >>>>>>>>>>>>>>>>>>>>> unwritten key groups (
> > >> > > >>>>>>>>>>>>>>> https://github.com/apache/flink/pull/13772)
> > >> > > >>>>>>>>>>>>>>>>>>>>>
> > >> > > >>>>>>>>>>>>>>>>>>>>> *Merged*
> > >> > > >>>>>>>>>>>>>>>>>>>>> - [FLINK-19154] Application mode deletes
> > >> > > >>> HA
> > >> > > >>>>> data
> > >> > > >>>>>>> in
> > >> > > >>>>>>>>>> case
> > >> > > >>>>>>>>>>> of
> > >> > > >>>>>>>>>>>>>>>>> suspended
> > >> > > >>>>>>>>>>>>>>>>>>>>> ZooKeeper connection
> > >> > > >>>>>>>>>>>>>>>>>>>>> - [FLINK-19569] Upgrade ICU4J to 67.1+
> > >> > > >>>>>>>>>>>>>>>>>>>>>
> > >> > > >>>>>>>>>>>>>>>>>>>>> Right now as it seems, progress is
> > >> > > >> mainly
> > >> > > >>>>>> blocked
> > >> > > >>>>>>>>> on a
> > >> > > >>>>>>>>>>>>> reviewer
> > >> > > >>>>>>>>>>>>>>>>> for
> > >> > > >>>>>>>>>>>>>>>>>>>>> FLINK-19717.
> > >> > > >>>>>>>>>>>>>>>>>>>>> Meanwhile, Xintong is keeping an eye on
> > >> > > >>> test
> > >> > > >>>>>>>>>>> instabilities
> > >> > > >>>>>>>>>>>>> [1]
> > >> > > >>>>>>>>>>>>>>> to
> > >> > > >>>>>>>>>>>>>>>>> see
> > >> > > >>>>>>>>>>>>>>>>>>> if
> > >> > > >>>>>>>>>>>>>>>>>>>>> there are any fixes that should be
> > >> > > >> applied
> > >> > > >>>> to
> > >> > > >>>>>>>>>>>> `release-1.11`.
> > >> > > >>>>>>>>>>>>>>>>>>>>>
> > >> > > >>>>>>>>>>>>>>>>>>>>> This is also a reminder, that if there
> > >> > > >> are
> > >> > > >>>>> other
> > >> > > >>>>>>>>>> blockers
> > >> > > >>>>>>>>>>>>> that
> > >> > > >>>>>>>>>>>>>>> we
> > >> > > >>>>>>>>>>>>>>>>>> need
> > >> > > >>>>>>>>>>>>>>>>>>> to
> > >> > > >>>>>>>>>>>>>>>>>>>>> be aware of, or a need to re-establish
> > >> > > >>>>> estimated
> > >> > > >>>>>>>>> time
> > >> > > >>>>>>>>>> for
> > >> > > >>>>>>>>>>>>>>> getting
> > >> > > >>>>>>>>>>>>>>>>>> fixes
> > >> > > >>>>>>>>>>>>>>>>>>>> in
> > >> > > >>>>>>>>>>>>>>>>>>>>> and delay the RC for 1.11.3, please do
> > >> > > >> let
> > >> > > >>>> us
> > >> > > >>>>>>> know!
> > >> > > >>>>>>>>>>>>>>>>>>>>>
> > >> > > >>>>>>>>>>>>>>>>>>>>> Cheers,
> > >> > > >>>>>>>>>>>>>>>>>>>>> Gordon
> > >> > > >>>>>>>>>>>>>>>>>>>>>
> > >> > > >>>>>>>>>>>>>>>>>>>>> [1]
> > >> > > >>>>>>>>>>>>>>>>>>>>>
> > >> > > >>>>>>>>>>>>>>>>>>>>>
> > >> > > >>>>>>>>>>>>>>>>>>>>
> > >> > > >>>>>>>>>>>>>>>>>>>
> > >> > > >>>>>>>>>>>>>>>>>>
> > >> > > >>>>>>>>>>>>>>>>>
> > >> > > >>>>>>>>>>>>>>>
> > >> > > >>>>>>>>>>>>>
> > >> > > >>>>>>>>>>>>
> > >> > > >>>>>>>>>>>
> > >> > > >>>>>>>>>>
> > >> > > >>>>>>>>>
> > >> > > >>>>>>>
> > >> > > >>>>>>
> > >> > > >>>>>
> > >> > > >>>>
> > >> > > >>>
> > >> > > >>
> > >> > >
> > >> >
> > >>
> >
> https://issues.apache.org/jira/issues/?jql=project%20%3D%20FLINK%20AND%20status%20in%20(Open%2C%20%22In%20Progress%22%2C%20Reopened)%20AND%20affectedVersion%20in%20(1.11.0%2C%201.11.1%2C%201.11.2)%20AND%20labels%20%3D%20test-stability
> > >> > > >>>>>>>>>>>>>>>>>>>>>
> > >> > > >>>>>>>>>>>>>>>>>>>>> On Mon, Oct 26, 2020 at 9:43 PM Kostas
> > >> > > >>>>> Kloudas <
> > >> > > >>>>>>>>>>>>>>>>> kkloudas@apache.org>
> > >> > > >>>>>>>>>>>>>>>>>>>>> wrote:
> > >> > > >>>>>>>>>>>>>>>>>>>>>
> > >> > > >>>>>>>>>>>>>>>>>>>>>> +1 for releasing Flink 1.11.3 as it
> > >> > > >>>>> contains a
> > >> > > >>>>>>>>> number
> > >> > > >>>>>>>>>>> of
> > >> > > >>>>>>>>>>>>>>>>> important
> > >> > > >>>>>>>>>>>>>>>>>>>>>> fixes and thanks Gordon and Xintong
> > >> > > >> for
> > >> > > >>>>>>>>> volunteering.
> > >> > > >>>>>>>>>>>>>>>>>>>>>>
> > >> > > >>>>>>>>>>>>>>>>>>>>>> Cheers,
> > >> > > >>>>>>>>>>>>>>>>>>>>>> Kostas
> > >> > > >>>>>>>>>>>>>>>>>>>>>>
> > >> > > >>>>>>>>>>>>>>>>>>>>>> On Mon, Oct 26, 2020 at 4:37 AM Yu Li
> > >> > > >> <
> > >> > > >>>>>>>>>>> carp84@gmail.com>
> > >> > > >>>>>>>>>>>>>>> wrote:
> > >> > > >>>>>>>>>>>>>>>>>>>>>>>
> > >> > > >>>>>>>>>>>>>>>>>>>>>>> +1 for releasing Flink 1.11.3, and
> > >> > > >>>> thanks
> > >> > > >>>>>>> Gordon
> > >> > > >>>>>>>>>> and
> > >> > > >>>>>>>>>>>>>>> Xintong
> > >> > > >>>>>>>>>>>>>>>>> for
> > >> > > >>>>>>>>>>>>>>>>>>>>>>> volunteering as our release
> > >> > > >> managers.
> > >> > > >>>>>>>>>>>>>>>>>>>>>>>
> > >> > > >>>>>>>>>>>>>>>>>>>>>>> Best Regards,
> > >> > > >>>>>>>>>>>>>>>>>>>>>>> Yu
> > >> > > >>>>>>>>>>>>>>>>>>>>>>>
> > >> > > >>>>>>>>>>>>>>>>>>>>>>>
> > >> > > >>>>>>>>>>>>>>>>>>>>>>> On Mon, 26 Oct 2020 at 09:45,
> > >> > > >> Xintong
> > >> > > >>>>> Song <
> > >> > > >>>>>>>>>>>>>>>>>> tonysong820@gmail.com>
> > >> > > >>>>>>>>>>>>>>>>>>>>>> wrote:
> > >> > > >>>>>>>>>>>>>>>>>>>>>>>
> > >> > > >>>>>>>>>>>>>>>>>>>>>>>> Thanks Gordan for starting this
> > >> > > >>>>>> discussion.
> > >> > > >>>>>>>>>>>>>>>>>>>>>>>> My pleasure to help with the
> > >> > > >> release
> > >> > > >>>>>>> process.
> > >> > > >>>>>>>>>>>>>>>>>>>>>>>>
> > >> > > >>>>>>>>>>>>>>>>>>>>>>>> Thank you~
> > >> > > >>>>>>>>>>>>>>>>>>>>>>>>
> > >> > > >>>>>>>>>>>>>>>>>>>>>>>> Xintong Song
> > >> > > >>>>>>>>>>>>>>>>>>>>>>>>
> > >> > > >>>>>>>>>>>>>>>>>>>>>>>>
> > >> > > >>>>>>>>>>>>>>>>>>>>>>>>
> > >> > > >>>>>>>>>>>>>>>>>>>>>>>> On Fri, Oct 23, 2020 at 11:29 PM
> > >> > > >>> Till
> > >> > > >>>>>>>>> Rohrmann <
> > >> > > >>>>>>>>>>>>>>>>>>>> trohrmann@apache.org
> > >> > > >>>>>>>>>>>>>>>>>>>>>>
> > >> > > >>>>>>>>>>>>>>>>>>>>>>>> wrote:
> > >> > > >>>>>>>>>>>>>>>>>>>>>>>>
> > >> > > >>>>>>>>>>>>>>>>>>>>>>>>> Thanks for starting this
> > >> > > >>> discussion
> > >> > > >>>>>>> Gordon.
> > >> > > >>>>>>>>>> There
> > >> > > >>>>>>>>>>>> are
> > >> > > >>>>>>>>>>>>>>> over
> > >> > > >>>>>>>>>>>>>>>>>> 100
> > >> > > >>>>>>>>>>>>>>>>>>>>> issues
> > >> > > >>>>>>>>>>>>>>>>>>>>>>>>> which are fixed for 1.11.3.
> > >> > > >> Hence
> > >> > > >>> +1
> > >> > > >>>>>> for a
> > >> > > >>>>>>>>>>> soonish
> > >> > > >>>>>>>>>>>>>>> 1.11.3
> > >> > > >>>>>>>>>>>>>>>>>>>> release.
> > >> > > >>>>>>>>>>>>>>>>>>>>>> Thanks
> > >> > > >>>>>>>>>>>>>>>>>>>>>>>>> for volunteering as our release
> > >> > > >>>>> managers
> > >> > > >>>>>>>>> Gordon
> > >> > > >>>>>>>>>>> and
> > >> > > >>>>>>>>>>>>>>>>> Xintong!
> > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > >> > > >>>>>>>>>>>>>>>>>>>>>>>>> Cheers,
> > >> > > >>>>>>>>>>>>>>>>>>>>>>>>> Till
> > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > >> > > >>>>>>>>>>>>>>>>>>>>>>>>> On Fri, Oct 23, 2020 at 5:02 PM
> > >> > > >>>> Tzu-Li
> > >> > > >>>>>>>>> (Gordon)
> > >> > > >>>>>>>>>>>> Tai <
> > >> > > >>>>>>>>>>>>>>>>>>>>>> tzulitai@apache.org
> > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > >> > > >>>>>>>>>>>>>>>>>>>>>>>>> wrote:
> > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>> Hi,
> > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>> Xintong and I would like to
> > >> > > >>> start a
> > >> > > >>>>>>>>> discussion
> > >> > > >>>>>>>>>>> for
> > >> > > >>>>>>>>>>>>>>>>> releasing
> > >> > > >>>>>>>>>>>>>>>>>>>> Flink
> > >> > > >>>>>>>>>>>>>>>>>>>>>>>> 1.11.3
> > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>> soon.
> > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>> It seems like we already have a
> > >> > > >>> few
> > >> > > >>>>>>>>> pressing
> > >> > > >>>>>>>>>>>> issues
> > >> > > >>>>>>>>>>>>>>> that
> > >> > > >>>>>>>>>>>>>>>>>> needs
> > >> > > >>>>>>>>>>>>>>>>>>>> to
> > >> > > >>>>>>>>>>>>>>>>>>>>> be
> > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>> included in a new hotfix
> > >> > > >> release:
> > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>>   - Heap-based timers’ restore
> > >> > > >>>>>> behaviour
> > >> > > >>>>>>>>> is
> > >> > > >>>>>>>>>>>>> causing a
> > >> > > >>>>>>>>>>>>>>>>>>> critical
> > >> > > >>>>>>>>>>>>>>>>>>>>>> recovery
> > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>>   issue for StateFun [1] [2]
> > >> > > >>> [3].
> > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>>   - There are several
> > >> > > >> robustness
> > >> > > >>>>>> issues
> > >> > > >>>>>>>>> for
> > >> > > >>>>>>>>>> the
> > >> > > >>>>>>>>>>>>>>> FLIP-27
> > >> > > >>>>>>>>>>>>>>>>> new
> > >> > > >>>>>>>>>>>>>>>>>>>>> source
> > >> > > >>>>>>>>>>>>>>>>>>>>>> API,
> > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>>   such as [4]. We already have
> > >> > > >>>> some
> > >> > > >>>>>>> users
> > >> > > >>>>>>>>>> using
> > >> > > >>>>>>>>>>>> the
> > >> > > >>>>>>>>>>>>>>>>> FLIP-27
> > >> > > >>>>>>>>>>>>>>>>>>> API
> > >> > > >>>>>>>>>>>>>>>>>>>>>> with
> > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>> 1.11.x,
> > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>>   so it would be important to
> > >> > > >>> get
> > >> > > >>>>>> those
> > >> > > >>>>>>>>> fixes
> > >> > > >>>>>>>>>>> in
> > >> > > >>>>>>>>>>>>> for
> > >> > > >>>>>>>>>>>>>>>>> 1.11.x
> > >> > > >>>>>>>>>>>>>>>>>>> as
> > >> > > >>>>>>>>>>>>>>>>>>>>>> well.
> > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>> Apart from the issues that are
> > >> > > >>>>> already
> > >> > > >>>>>>>>> marked
> > >> > > >>>>>>>>>> as
> > >> > > >>>>>>>>>>>>>>> blocker
> > >> > > >>>>>>>>>>>>>>>>> for
> > >> > > >>>>>>>>>>>>>>>>>>>>> 1.11.3
> > >> > > >>>>>>>>>>>>>>>>>>>>>> in
> > >> > > >>>>>>>>>>>>>>>>>>>>>>>> our
> > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>> JIRA [5], please let us know in
> > >> > > >>>> this
> > >> > > >>>>>>>>> thread if
> > >> > > >>>>>>>>>>>> there
> > >> > > >>>>>>>>>>>>>>> is
> > >> > > >>>>>>>>>>>>>>>>>>> already
> > >> > > >>>>>>>>>>>>>>>>>>>>>> ongoing
> > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>> work for other important fixes
> > >> > > >>> that
> > >> > > >>>>> we
> > >> > > >>>>>>>>> should
> > >> > > >>>>>>>>>>> try
> > >> > > >>>>>>>>>>>> to
> > >> > > >>>>>>>>>>>>>>>>>> include.
> > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>> Xintong and I would like to
> > >> > > >>>> volunteer
> > >> > > >>>>>> for
> > >> > > >>>>>>>>>>> managing
> > >> > > >>>>>>>>>>>>>>> this
> > >> > > >>>>>>>>>>>>>>>>>>> release,
> > >> > > >>>>>>>>>>>>>>>>>>>>> and
> > >> > > >>>>>>>>>>>>>>>>>>>>>>>> will
> > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>> try to communicate the priority
> > >> > > >>> of
> > >> > > >>>>>>> pending
> > >> > > >>>>>>>>>>>> blockers
> > >> > > >>>>>>>>>>>>>>> over
> > >> > > >>>>>>>>>>>>>>>>> the
> > >> > > >>>>>>>>>>>>>>>>>>>> next
> > >> > > >>>>>>>>>>>>>>>>>>>>>> few
> > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>> days.
> > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>> Since the aforementioned issues
> > >> > > >>> are
> > >> > > >>>>>> quite
> > >> > > >>>>>>>>>>>> critical,
> > >> > > >>>>>>>>>>>>>>> we’d
> > >> > > >>>>>>>>>>>>>>>>>> like
> > >> > > >>>>>>>>>>>>>>>>>>> to
> > >> > > >>>>>>>>>>>>>>>>>>>>> aim
> > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>> for a *feature
> > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>> freeze by the end of next week
> > >> > > >>>> (Oct.
> > >> > > >>>>>>> 30th)*
> > >> > > >>>>>>>>>> and
> > >> > > >>>>>>>>>>>>> start
> > >> > > >>>>>>>>>>>>>>> the
> > >> > > >>>>>>>>>>>>>>>>>>>> release
> > >> > > >>>>>>>>>>>>>>>>>>>>>> voting
> > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>> process the week after.
> > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>> If that is too short of a
> > >> > > >> notice
> > >> > > >>>> and
> > >> > > >>>>>> you
> > >> > > >>>>>>>>> might
> > >> > > >>>>>>>>>>>> need
> > >> > > >>>>>>>>>>>>>>> more
> > >> > > >>>>>>>>>>>>>>>>>> time,
> > >> > > >>>>>>>>>>>>>>>>>>>>>> please
> > >> > > >>>>>>>>>>>>>>>>>>>>>>>> let
> > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>> us know!
> > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>> Cheers,
> > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>> Gordon
> > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>> [1]
> > >> > > >>>>>>>>>>>>> https://issues.apache.org/jira/browse/FLINK-19692
> > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>> [2]
> > >> > > >>>>>>>>>>>>> https://issues.apache.org/jira/browse/FLINK-19741
> > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>> [3]
> > >> > > >>>>>>>>>>>>> https://issues.apache.org/jira/browse/FLINK-19748
> > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>> [4]
> > >> > > >>>>>>>>>>>>> https://issues.apache.org/jira/browse/FLINK-19717
> > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>> [5]
> > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > >> > > >>>>>>>>>>>>>>>>>>>>>>>>
> > >> > > >>>>>>>>>>>>>>>>>>>>>>
> > >> > > >>>>>>>>>>>>>>>>>>>>>
> > >> > > >>>>>>>>>>>>>>>>>>>>
> > >> > > >>>>>>>>>>>>>>>>>>>
> > >> > > >>>>>>>>>>>>>>>>>>
> > >> > > >>>>>>>>>>>>>>>>>
> > >> > > >>>>>>>>>>>>>>>
> > >> > > >>>>>>>>>>>>>
> > >> > > >>>>>>>>>>>>
> > >> > > >>>>>>>>>>>
> > >> > > >>>>>>>>>>
> > >> > > >>>>>>>>>
> > >> > > >>>>>>>
> > >> > > >>>>>>
> > >> > > >>>>>
> > >> > > >>>>
> > >> > > >>>
> > >> > > >>
> > >> > >
> > >> >
> > >>
> >
> https://issues.apache.org/jira/issues/?jql=project%20%3D%20FLINK%20AND%20priority%20%3D%20Blocker%20AND%20fixVersion%20%3D%201.11.3
> > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > >> > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > >> > > >>>>>>>>>>>>>>>>>>>>>>>>
> > >> > > >>>>>>>>>>>>>>>>>>>>>>
> > >> > > >>>>>>>>>>>>>>>>>>>>>
> > >> > > >>>>>>>>>>>>>>>>>>>>
> > >> > > >>>>>>>>>>>>>>>>>>>
> > >> > > >>>>>>>>>>>>>>>>>>
> > >> > > >>>>>>>>>>>>>>>>>>
> > >> > > >>>>>>>>>>>>>>>>>> --
> > >> > > >>>>>>>>>>>>>>>>>> Best, Jingsong Lee
> > >> > > >>>>>>>>>>>>>>>>>>
> > >> > > >>>>>>>>>>>>>>>>>
> > >> > > >>>>>>>>>>>>>>>>
> > >> > > >>>>>>>>>>>>>>>
> > >> > > >>>>>>>>>>>>>>
> > >> > > >>>>>>>>>>>>>
> > >> > > >>>>>>>>>>>>
> > >> > > >>>>>>>>>>>
> > >> > > >>>>>>>>>>
> > >> > > >>>>>>>>>
> > >> > > >>>>>>>>
> > >> > > >>>>>>>
> > >> > > >>>>>>
> > >> > > >>>>>
> > >> > > >>>>
> > >> > > >>>
> > >> > > >>
> > >> > >
> > >> > >
> > >> > >
> > >> >
> > >>
> > >
> >
>

Re: [DISCUSS] Releasing Apache Flink 1.11.3

Posted by Gyula Fóra <gy...@gmail.com>.
Hi All!

I have found the following issue today which might be considered a blocker
for this release as well:

https://issues.apache.org/jira/browse/FLINK-20221

Could someone please quickly provide a second set of eyes and confirm that
this is indeed a big problem? :)

Thank you!
Gyula

On Wed, Nov 18, 2020 at 5:12 AM Xintong Song <to...@gmail.com> wrote:

> Hi devs,
>
> Updates on the progress of preparing the 1.11.3 release.
>
> We are approaching the creation of our first release candidate. Thanks all
> for the efforts so far.
>
> # Ongoing threads
>
>    - *FLIP-27 backportings:* 74 out of 102 commits are already backported.
>    Stephan and Becket are actively working on this.
>    - *Blockers:* There are 2 remaining blockers, both have been fixed on
>    the master branch and should be backported along with the FLIP-27
> changes.
>
> As soon as the above threads are done, we will create our first release
> candidate. If there's any other issues that you believe should be a release
> blocker, please reach out to either me or Gordon.
>
> # JIRA version
>
> Version 1.11.4 has been created on JIRA. I've already tagged some of the
> tickets with the new fix version. Please feel free to continue with the
> efforts and merge once they are ready. I'll double check and update the fix
> versions before the release.
>
> Thank you~
>
> Xintong Song
>
>
>
> On Thu, Nov 12, 2020 at 1:31 PM Xintong Song <to...@gmail.com>
> wrote:
>
> > Thanks for the notice and fix, Roman.
> >
> > Thank you~
> >
> > Xintong Song
> >
> >
> >
> > On Wed, Nov 11, 2020 at 5:53 PM Khachatryan Roman <
> > khachatryan.roman@gmail.com> wrote:
> >
> >> Hi,
> >>
> >> I'd like FLINK-20079 [1] to be merged into 1.11 and included in 1.11.3.
> >>
> >> [1] https://issues.apache.org/jira/browse/FLINK-20079
> >>
> >> Regards,
> >> Roman
> >>
> >>
> >> On Tue, Nov 10, 2020 at 8:21 AM Xintong Song <to...@gmail.com>
> >> wrote:
> >>
> >> > Thanks for the notice, Dian.
> >> >
> >> > Thank you~
> >> >
> >> > Xintong Song
> >> >
> >> >
> >> >
> >> > On Tue, Nov 10, 2020 at 10:19 AM Dian Fu <di...@gmail.com>
> wrote:
> >> >
> >> > > Hi Xintong,
> >> > >
> >> > > I want to bring one more issue to your attention [1]. The test case
> >> > > UnalignedCheckpointCompatibilityITCase.test failed several times in
> >> the
> >> > > last nightly test of release-1.11. We need to figure out if it's
> just
> >> an
> >> > > instable test or caused by recent changes.
> >> > >
> >> > > [1] https://issues.apache.org/jira/browse/FLINK-20065
> >> > >
> >> > > > 在 2020年11月10日，上午9:24，Xintong Song <to...@gmail.com> 写道：
> >> > > >
> >> > > > Thanks for the replies.
> >> > > >
> >> > > > Thank you~
> >> > > >
> >> > > > Xintong Song
> >> > > >
> >> > > >
> >> > > >
> >> > > > On Tue, Nov 10, 2020 at 1:09 AM Becket Qin <be...@gmail.com>
> >> > wrote:
> >> > > >
> >> > > >> Hi Xintong,
> >> > > >>
> >> > > >> Thanks for driving the release. Just want to sync up on the
> FLIP-27
> >> > > >> backporting. Stephan and I are still trying to backport a bunch
> of
> >> > > patches
> >> > > >> of Source to 1.11.3. Including:
> >> > > >>
> >> > > >> [FLINK-19698][connector/common] Add a close() method to the
> >> > SplitReader.
> >> > > >> [FLINK-19717] SourceReaderBase.pollNext may return END_OF_INPUT
> if
> >> > > >> SplitReader.fetch throws
> >> > > >> [FLINK-19535] [connector/common] Avoid failing a job multiple
> >> times in
> >> > > >> SourceCoordinator.
> >> > > >> [FLINK-19265] [FLINK-20049][core] Source API final adjustments.
> >> > > >>
> >> > > >> and a few more fixes.
> >> > > >>
> >> > > >> We are currently trying to fix them in 1.12 first so it might
> take
> >> a
> >> > > little
> >> > > >> longer to backport them to 1.11.3. I think it will probably take
> >> us a
> >> > > few
> >> > > >> more days to finish the backport. So that would roughly be the
> end
> >> of
> >> > > this
> >> > > >> week.
> >> > > >>
> >> > > >> Thanks,
> >> > > >>
> >> > > >> Jiangjie (Becket) Qin
> >> > > >>
> >> > > >>
> >> > > >>
> >> > > >>
> >> > > >> On Mon, Nov 9, 2020 at 9:57 PM Till Rohrmann <
> trohrmann@apache.org
> >> >
> >> > > wrote:
> >> > > >>
> >> > > >>> Yes, I've downgraded FLINK-19816 to critical.
> >> > > >>>
> >> > > >>> Cheers,
> >> > > >>> Till
> >> > > >>>
> >> > > >>> On Mon, Nov 9, 2020 at 10:19 AM Xintong Song <
> >> tonysong820@gmail.com>
> >> > > >>> wrote:
> >> > > >>>
> >> > > >>>> Thanks for the notice, Till.
> >> > > >>>>
> >> > > >>>> I just checked and found FLINK-20033 is already fixed. Shall we
> >> also
> >> > > >>>> downgrade FLINK-19816 to `Critical`?
> >> > > >>>>
> >> > > >>>> Thank you~
> >> > > >>>>
> >> > > >>>> Xintong Song
> >> > > >>>>
> >> > > >>>>
> >> > > >>>>
> >> > > >>>> On Mon, Nov 9, 2020 at 4:42 PM Till Rohrmann <
> >> trohrmann@apache.org>
> >> > > >>> wrote:
> >> > > >>>>
> >> > > >>>>> I would like to bring one more critical issue to your
> attention
> >> > which
> >> > > >>> is
> >> > > >>>>> FLINK-20033 [1]. I believe that this issue is actually causing
> >> what
> >> > > >> has
> >> > > >>>>> been reported in FLINK-19816 [2]. I hope to have it fixed by
> the
> >> > end
> >> > > >> of
> >> > > >>>>> today. Once FLINK-20033 is fixed, I think that we don't have
> to
> >> > block
> >> > > >>> the
> >> > > >>>>> release on FLINK-19816.
> >> > > >>>>>
> >> > > >>>>> [1] https://issues.apache.org/jira/browse/FLINK-20033
> >> > > >>>>> [2] https://issues.apache.org/jira/browse/FLINK-19816
> >> > > >>>>>
> >> > > >>>>> Cheers,
> >> > > >>>>> Till
> >> > > >>>>>
> >> > > >>>>> On Mon, Nov 9, 2020 at 4:05 AM Xintong Song <
> >> tonysong820@gmail.com
> >> > >
> >> > > >>>> wrote:
> >> > > >>>>>
> >> > > >>>>>> Hi devs,
> >> > > >>>>>>
> >> > > >>>>>> I'd like to provide an update on the progress of preparing
> >> release
> >> > > >>>>> 1.11.3.
> >> > > >>>>>>
> >> > > >>>>>> *Blockers*
> >> > > >>>>>> We currently have 3 remaining blockers. (3 resolved and 1
> >> emerged
> >> > > >>>>> compared
> >> > > >>>>>> to last week)
> >> > > >>>>>>
> >> > > >>>>>> - [FLINK-19698] Add close() method and onCheckpointComplete()
> >> to
> >> > > >> the
> >> > > >>>>>> Source.
> >> > > >>>>>> The issue has been fixed on the master branch. It's currently
> >> > > >> blocked
> >> > > >>>> on
> >> > > >>>>>> the FLIP-27 backportings to backport it to the 1.11 branch.
> >> > > >>>>>>
> >> > > >>>>>> - [FLINK-19717] SourceReaderBase.pollNext may return
> >> END_OF_INPUT
> >> > > >> if
> >> > > >>>>>> SplitReader.fetch throws
> >> > > >>>>>> A PR has been opened and reviewed. From the discussions on
> the
> >> PR,
> >> > > >> it
> >> > > >>>>> looks
> >> > > >>>>>> close to mergeable.
> >> > > >>>>>>
> >> > > >>>>>> - [FLINK-19816] Flink restored from a wrong checkpoint (a
> very
> >> old
> >> > > >>> one
> >> > > >>>>> and
> >> > > >>>>>> not the last completed one)
> >> > > >>>>>> This is a newly emerged blocker and Matthias is working on
> it.
> >> > > >>>>>>
> >> > > >>>>>> *Test Instabilities*
> >> > > >>>>>> We currently have 27 test instabilities[1].
> >> > > >>>>>> AFAIK, none of them are as serious as to block the 1.11.3
> >> release.
> >> > > >>>>>>
> >> > > >>>>>> *FLIP-27 Backprotings*
> >> > > >>>>>>
> >> > > >>>>>> I noticed that there's no jira issues opened on the FLIP-27
> >> > > >>> backporting
> >> > > >>>>>> efforts, which is part of the major efforts planned for the
> >> 1.11.3
> >> > > >>>>> release,
> >> > > >>>>>> making it hard to track the progress.
> >> > > >>>>>>
> >> > > >>>>>>
> >> > > >>>>>> @Stephan and @Becket, could you please share the updates on
> the
> >> > > >>>>> backporting
> >> > > >>>>>> efforts? How is the progress and when are the efforts
> expected
> >> to
> >> > > >> be
> >> > > >>>>>> finished? It would be appreciated and helpful if we can have
> a
> >> > jira
> >> > > >>>>> ticket
> >> > > >>>>>> to track the backporing progress publicly.
> >> > > >>>>>>
> >> > > >>>>>>
> >> > > >>>>>> Thank you~
> >> > > >>>>>>
> >> > > >>>>>> Xintong Song
> >> > > >>>>>>
> >> > > >>>>>>
> >> > > >>>>>> [1]
> >> > > >>>>>>
> >> > > >>>>>>
> >> > > >>>>>
> >> > > >>>>
> >> > > >>>
> >> > > >>
> >> > >
> >> >
> >>
> https://issues.apache.org/jira/browse/FLINK-16908?filter=12348580&jql=project%20%3D%20FLINK%20AND%20resolution%20%3D%20Unresolved%20AND%20affectedVersion%20in%20(1.11.0%2C%201.11.1%2C%201.11.2%2C%201.11.3)%20AND%20labels%20%3D%20test-stability%20ORDER%20BY%20updated%20DESC%2C%20created%20DESC
> >> > > >>>>>>
> >> > > >>>>>> On Wed, Nov 4, 2020 at 3:58 PM Tzu-Li (Gordon) Tai <
> >> > > >>>> tzulitai@apache.org>
> >> > > >>>>>> wrote:
> >> > > >>>>>>
> >> > > >>>>>>> Thanks for the efforts so far with pushing for 1.11.3.
> >> > > >>>>>>>
> >> > > >>>>>>> I'd like to provide a mid-week update on how we're looking
> >> with
> >> > > >> all
> >> > > >>>> the
> >> > > >>>>>>> pending blockers and backports:
> >> > > >>>>>>>
> >> > > >>>>>>> *Blockers**:*
> >> > > >>>>>>>
> >> > > >>>>>>> Currently all blockers either have an open PR, or have been
> >> > > >> merged.
> >> > > >>>>> I'll
> >> > > >>>>>>> highlight below the blockers *that still require reviewing
> >> > > >> efforts
> >> > > >>>> for
> >> > > >>>>>> them
> >> > > >>>>>>> to move forward:*
> >> > > >>>>>>>
> >> > > >>>>>>> - [FLINK-19909] Flink application in attach mode could not
> >> > > >>> terminate
> >> > > >>>>> when
> >> > > >>>>>>> the only job is canceled. PR:
> >> > > >>>>> https://github.com/apache/flink/pull/13911
> >> > > >>>>>>> - [FLINK-19717] SourceReaderBase.pollNext may return
> >> END_OF_INPUT
> >> > > >>> if
> >> > > >>>>>>> SplitReader.fetch throws. PR:
> >> > > >>>>> https://github.com/apache/flink/pull/13776
> >> > > >>>>>>>
> >> > > >>>>>>> The above PRs currently have no reviews at all yet. They
> seem
> >> to
> >> > > >>>>> already
> >> > > >>>>>>> have designated reviewers.
> >> > > >>>>>>>
> >> > > >>>>>>>
> >> > > >>>>>>> *Backports:*
> >> > > >>>>>>> - FLIP-27 robustness improvement backports: Stephan is
> >> currently
> >> > > >>>>> working
> >> > > >>>>>> on
> >> > > >>>>>>> backporting several FLIP-27 changes. There are no PRs yet
> for
> >> the
> >> > > >>>>>>> backported changes.
> >> > > >>>>>>> - DataStreamUtils.collect() refactorings backport: Steven Wu
> >> > > >>>> mentioned
> >> > > >>>>> to
> >> > > >>>>>>> backport this, but AFAIK this isn't assigned to anyone yet.
> >> > > >>>>>>>
> >> > > >>>>>>> *ETAs*:
> >> > > >>>>>>>
> >> > > >>>>>>> I'd like to request ETAs for the remaining backports, to
> >> prevent
> >> > > >> a
> >> > > >>>>> creep
> >> > > >>>>>> in
> >> > > >>>>>>> the scope of this bugfix release.
> >> > > >>>>>>>
> >> > > >>>>>>> We already have Flink users that would benefit from fixes
> that
> >> > > >> have
> >> > > >>>>> been
> >> > > >>>>>>> merged to release-1.11 already, so technically speaking the
> >> > > >>> backports
> >> > > >>>>>>> should be considered "nice-to-have" (to the best of my
> >> knowledge
> >> > > >> of
> >> > > >>>> the
> >> > > >>>>>>> changes) and could potentially be moved to a follow-up
> 1.11.4.
> >> > > >>>>>>> Most notably, the Stateful Functions project is already
> >> waiting
> >> > > >> on
> >> > > >>>>> Flink
> >> > > >>>>>>> 1.11.3 to address critical recovery issues (please see the
> >> > > >> StateFun
> >> > > >>>>> 2.2.1
> >> > > >>>>>>> release discussion thread [1]).
> >> > > >>>>>>>
> >> > > >>>>>>> @Stephan Ewen <se...@apache.org> @Becket Qin <
> >> > > >> becket.qin@gmail.com
> >> > > >>>>
> >> > > >>>>>> could
> >> > > >>>>>>> you provide an ETA for the FLIP-27 backports? It would help
> to
> >> > > >> get
> >> > > >>> a
> >> > > >>>>>> better
> >> > > >>>>>>> estimate to decide how we proceed here.
> >> > > >>>>>>>
> >> > > >>>>>>> Cheers,
> >> > > >>>>>>> Gordon
> >> > > >>>>>>>
> >> > > >>>>>>> [1]
> >> > > >>>>>>>
> >> > > >>>>>>>
> >> > > >>>>>>
> >> > > >>>>>
> >> > > >>>>
> >> > > >>>
> >> > > >>
> >> > >
> >> >
> >>
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Releasing-StateFun-hotfix-version-2-2-1-td46239.html
> >> > > >>>>>>>
> >> > > >>>>>>> On Wed, Nov 4, 2020 at 3:16 PM Tzu-Li (Gordon) Tai <
> >> > > >>>>> tzulitai@apache.org>
> >> > > >>>>>>> wrote:
> >> > > >>>>>>>
> >> > > >>>>>>>>> The collect() utils can be picked back, I see no issue
> with
> >> > > >>> that
> >> > > >>>>> (it
> >> > > >>>>>> is
> >> > > >>>>>>>> isolated utilities).
> >> > > >>>>>>>>
> >> > > >>>>>>>> Just checking on all the requested backports mentioned in
> >> this
> >> > > >>>>> thread,
> >> > > >>>>>>> and
> >> > > >>>>>>>> figuring out which ones seem to still be unassigned / open.
> >> > > >>>>>>>>
> >> > > >>>>>>>> Is someone working on backporting
> >> > > >>>>>>>>
> >> > > >>>>>>>
> >> > > >>>>>>
> >> > > >>>>>
> >> > > >>>>
> >> > > >>>
> >> > > >>
> >> > >
> >> >
> >>
> https://github.com/apache/flink/commit/09a7a66b7313fea64817fe960a8da1265b428efc
> >> > > >>>>>>>> to release-1.11 at the moment?
> >> > > >>>>>>>>
> >> > > >>>>>>>> On Wed, Nov 4, 2020 at 1:39 AM Steven Wu <
> >> stevenz3wu@gmail.com
> >> > > >>>
> >> > > >>>>> wrote:
> >> > > >>>>>>>>
> >> > > >>>>>>>>> @Stephan Ewen <se...@apache.org> yeah, we can do that.
> >> don't
> >> > > >>>> worry
> >> > > >>>>>>> about
> >> > > >>>>>>>>> it. your earlier email had the perfect explanation on why
> >> file
> >> > > >>>>> source
> >> > > >>>>>>>>> shouldn't be backported.
> >> > > >>>>>>>>>
> >> > > >>>>>>>>> On Tue, Nov 3, 2020 at 3:37 AM Stephan Ewen <
> >> sewen@apache.org
> >> > > >>>
> >> > > >>>>> wrote:
> >> > > >>>>>>>>>
> >> > > >>>>>>>>>> @Steven would it be possible to initially copy some of
> the
> >> > > >>> code
> >> > > >>>>> into
> >> > > >>>>>>> the
> >> > > >>>>>>>>>> iceberg source and later replace it by a dependency on
> the
> >> > > >>> Flink
> >> > > >>>>>> file
> >> > > >>>>>>>>>> source?
> >> > > >>>>>>>>>>
> >> > > >>>>>>>>>> On Mon, Nov 2, 2020 at 8:33 PM Steven Wu <
> >> > > >>> stevenz3wu@gmail.com>
> >> > > >>>>>>> wrote:
> >> > > >>>>>>>>>>
> >> > > >>>>>>>>>>> Stephan, thanks a lot for explaining the file connector.
> >> > > >>> that
> >> > > >>>>>> makes
> >> > > >>>>>>>>>> sense.
> >> > > >>>>>>>>>>>
> >> > > >>>>>>>>>>> I was asking because we were trying to reuse some of the
> >> > > >>>>>>>>> implementations
> >> > > >>>>>>>>>> in
> >> > > >>>>>>>>>>> the file source for Iceberg source. Flink Iceberg source
> >> > > >>> lives
> >> > > >>>>> in
> >> > > >>>>>>> the
> >> > > >>>>>>>>>>> Iceberg repo, which is not possible to code against the
> >> > > >>> master
> >> > > >>>>>>> branch
> >> > > >>>>>>>>> of
> >> > > >>>>>>>>>>> the Flink code.
> >> > > >>>>>>>>>>>
> >> > > >>>>>>>>>>> On Mon, Nov 2, 2020 at 3:31 AM Stephan Ewen <
> >> > > >>> sewen@apache.org
> >> > > >>>>>
> >> > > >>>>>>> wrote:
> >> > > >>>>>>>>>>>
> >> > > >>>>>>>>>>>> Hi Steven!
> >> > > >>>>>>>>>>>>
> >> > > >>>>>>>>>>>> So far there are no plans to pick back the file system
> >> > > >>>>> connector
> >> > > >>>>>>>>> code.
> >> > > >>>>>>>>>>> This
> >> > > >>>>>>>>>>>> is still evolving and not finalized for 1.12, so I
> don't
> >> > > >>>> feel
> >> > > >>>>> it
> >> > > >>>>>>> is
> >> > > >>>>>>>>> a
> >> > > >>>>>>>>>>> good
> >> > > >>>>>>>>>>>> candidate to be backported.
> >> > > >>>>>>>>>>>> However, with the base connector changes backported,
> you
> >> > > >>>>> should
> >> > > >>>>>> be
> >> > > >>>>>>>>> able
> >> > > >>>>>>>>>>> to
> >> > > >>>>>>>>>>>> run the file connector code from master against 1.11.3.
> >> > > >>>>>>>>>>>>
> >> > > >>>>>>>>>>>> The collect() utils can be picked back, I see no issue
> >> > > >>> with
> >> > > >>>>> that
> >> > > >>>>>>>>> (it is
> >> > > >>>>>>>>>>>> isolated utilities).
> >> > > >>>>>>>>>>>>
> >> > > >>>>>>>>>>>> Best,
> >> > > >>>>>>>>>>>> Stephan
> >> > > >>>>>>>>>>>>
> >> > > >>>>>>>>>>>>
> >> > > >>>>>>>>>>>> On Mon, Nov 2, 2020 at 3:02 AM Steven Wu <
> >> > > >>>>> stevenz3wu@gmail.com>
> >> > > >>>>>>>>> wrote:
> >> > > >>>>>>>>>>>>
> >> > > >>>>>>>>>>>>> Basically, it would be great to get the latest code in
> >> > > >>> the
> >> > > >>>>>>>>>>>>> flink-connector-files (FLIP-27).
> >> > > >>>>>>>>>>>>>
> >> > > >>>>>>>>>>>>> On Sat, Oct 31, 2020 at 9:57 AM Steven Wu <
> >> > > >>>>>> stevenz3wu@gmail.com
> >> > > >>>>>>>>
> >> > > >>>>>>>>>>> wrote:
> >> > > >>>>>>>>>>>>>
> >> > > >>>>>>>>>>>>>> Stephan, it will be great if we can also backport
> >> > > >> the
> >> > > >>>>>>>>>> DataStreamUtils
> >> > > >>>>>>>>>>>>>> related commits that help with collecting output
> >> > > >> from
> >> > > >>>>>>> unbounded
> >> > > >>>>>>>>>>>> streams.
> >> > > >>>>>>>>>>>>>> e.g.
> >> > > >>>>>>>>>>>>>>
> >> > > >>>>>>>>>>>>>>
> >> > > >>>>>>>>>>>>>>
> >> > > >>>>>>>>>>>>>
> >> > > >>>>>>>>>>>>
> >> > > >>>>>>>>>>>
> >> > > >>>>>>>>>>
> >> > > >>>>>>>>>
> >> > > >>>>>>>
> >> > > >>>>>>
> >> > > >>>>>
> >> > > >>>>
> >> > > >>>
> >> > > >>
> >> > >
> >> >
> >>
> https://github.com/apache/flink/commit/09a7a66b7313fea64817fe960a8da1265b428efc
> >> > > >>>>>>>>>>>>>>
> >> > > >>>>>>>>>>>>>> I tried to copy and paste the code to unblock
> >> > > >> myself.
> >> > > >>>> but
> >> > > >>>>> it
> >> > > >>>>>>>>>> quickly
> >> > > >>>>>>>>>>>> got
> >> > > >>>>>>>>>>>>>> into the rabbit hole of more and more code.
> >> > > >>>>>>>>>>>>>>
> >> > > >>>>>>>>>>>>>> On Fri, Oct 30, 2020 at 11:02 AM Stephan Ewen <
> >> > > >>>>>>> sewen@apache.org
> >> > > >>>>>>>>>>
> >> > > >>>>>>>>>>>> wrote:
> >> > > >>>>>>>>>>>>>>
> >> > > >>>>>>>>>>>>>>> I have started with backporting the source API
> >> > > >>> changes.
> >> > > >>>>>> Some
> >> > > >>>>>>>>> minor
> >> > > >>>>>>>>>>>>>>> conflicts to solve, will need a bit more to finish
> >> > > >>>> this.
> >> > > >>>>>>>>>>>>>>>
> >> > > >>>>>>>>>>>>>>> On Fri, Oct 30, 2020 at 7:25 AM Tzu-Li (Gordon)
> >> > > >> Tai <
> >> > > >>>>>>>>>>>>> tzulitai@apache.org>
> >> > > >>>>>>>>>>>>>>> wrote:
> >> > > >>>>>>>>>>>>>>>
> >> > > >>>>>>>>>>>>>>>> @Stephan Ewen <se...@apache.org>
> >> > > >>>>>>>>>>>>>>>> Are there already plans or ongoing efforts for
> >> > > >>>>>> backporting
> >> > > >>>>>>>>> the
> >> > > >>>>>>>>>>> list
> >> > > >>>>>>>>>>>> of
> >> > > >>>>>>>>>>>>>>>> FLIP-27 changes that you posted?
> >> > > >>>>>>>>>>>>>>>>
> >> > > >>>>>>>>>>>>>>>> On Thu, Oct 29, 2020 at 7:08 PM Xintong Song <
> >> > > >>>>>>>>>>> tonysong820@gmail.com
> >> > > >>>>>>>>>>>>>
> >> > > >>>>>>>>>>>>>>>> wrote:
> >> > > >>>>>>>>>>>>>>>>
> >> > > >>>>>>>>>>>>>>>>> Hi folks,
> >> > > >>>>>>>>>>>>>>>>>
> >> > > >>>>>>>>>>>>>>>>> Just to provide some updates concerning the
> >> > > >> status
> >> > > >>>> on
> >> > > >>>>>> the
> >> > > >>>>>>>>>>>>>>>>> test instabilities.
> >> > > >>>>>>>>>>>>>>>>>
> >> > > >>>>>>>>>>>>>>>>> Currently, we have 30 unresolved tickets labeled
> >> > > >>>> with
> >> > > >>>>>>>>> `Affects
> >> > > >>>>>>>>>>>>> Version`
> >> > > >>>>>>>>>>>>>>>>> 1.11.x.
> >> > > >>>>>>>>>>>>>>>>>
> >> > > >>>>>>>>>>>>>>>>>
> >> > > >>>>>>>>>>>>>>>
> >> > > >>>>>>>>>>>>>
> >> > > >>>>>>>>>>>>
> >> > > >>>>>>>>>>>
> >> > > >>>>>>>>>>
> >> > > >>>>>>>>>
> >> > > >>>>>>>
> >> > > >>>>>>
> >> > > >>>>>
> >> > > >>>>
> >> > > >>>
> >> > > >>
> >> > >
> >> >
> >>
> https://issues.apache.org/jira/browse/FLINK-19775?filter=12348580&jql=project%20%3D%20FLINK%20AND%20resolution%20%3D%20Unresolved%20AND%20affectedVersion%20in%20(1.11.0%2C%201.11.1%2C%201.11.2%2C%201.11.3)%20AND%20labels%20%3D%20test-stability%20ORDER%20BY%20created%20DESC
> >> > > >>>>>>>>>>>>>>>>>
> >> > > >>>>>>>>>>>>>>>>> Among the 30 tickets, 11 of them are:
> >> > > >>>>>>>>>>>>>>>>> - Have occured in the recent 3 months
> >> > > >>>>>>>>>>>>>>>>> - Not confirmed to be pure testability issues
> >> > > >>>>>>>>>>>>>>>>> - Not confirmed to be rare condition cases
> >> > > >>>>>>>>>>>>>>>>>
> >> > > >>>>>>>>>>>>>>>>> It would be nice if someone familiar with these
> >> > > >>>>>> components
> >> > > >>>>>>>>> can
> >> > > >>>>>>>>>>>> take a
> >> > > >>>>>>>>>>>>>>> look
> >> > > >>>>>>>>>>>>>>>>> into these issues.
> >> > > >>>>>>>>>>>>>>>>>
> >> > > >>>>>>>>>>>>>>>>> -
> >> > > >>> https://issues.apache.org/jira/browse/FLINK-17159
> >> > > >>>>>> (ES6)
> >> > > >>>>>>>>>>>>>>>>> -
> >> > > >>> https://issues.apache.org/jira/browse/FLINK-17912
> >> > > >>>>>>> (Kafka)
> >> > > >>>>>>>>>>>>>>>>> -
> >> > > >>> https://issues.apache.org/jira/browse/FLINK-17949
> >> > > >>>>>>> (Kafka)
> >> > > >>>>>>>>>>>>>>>>> ⁃
> >> > > >>> https://issues.apache.org/jira/browse/FLINK-18444
> >> > > >>>>>>> (Kafka)
> >> > > >>>>>>>>>>>>>>>>> -
> >> > > >>> https://issues.apache.org/jira/browse/FLINK-18634
> >> > > >>>>>>> (Kafka)
> >> > > >>>>>>>>>>>>>>>>> -
> >> > > >>> https://issues.apache.org/jira/browse/FLINK-18648
> >> > > >>>>>>> (Kafka)
> >> > > >>>>>>>>>>>>>>>>> -
> >> > > >>> https://issues.apache.org/jira/browse/FLINK-18807
> >> > > >>>>>>> (Kafka)
> >> > > >>>>>>>>>>>>>>>>> -
> >> > > >>> https://issues.apache.org/jira/browse/FLINK-19369
> >> > > >>>>>>>>>>>> (BlobClientTest)
> >> > > >>>>>>>>>>>>>>>>> -
> >> > > >>> https://issues.apache.org/jira/browse/FLINK-19436
> >> > > >>>>>>> (TPCDS)
> >> > > >>>>>>>>>>>>>>>>> -
> >> > > >>> https://issues.apache.org/jira/browse/FLINK-19690
> >> > > >>>>>>>>>>>> (Format/Parquet)
> >> > > >>>>>>>>>>>>>>>>> -
> >> > > >>> https://issues.apache.org/jira/browse/FLINK-19775
> >> > > >>>>>>>>>>>>>>>>> (SystemProcessingTimeServiceTest)
> >> > > >>>>>>>>>>>>>>>>>
> >> > > >>>>>>>>>>>>>>>>> Thank you~
> >> > > >>>>>>>>>>>>>>>>>
> >> > > >>>>>>>>>>>>>>>>> Xintong Song
> >> > > >>>>>>>>>>>>>>>>>
> >> > > >>>>>>>>>>>>>>>>>
> >> > > >>>>>>>>>>>>>>>>>
> >> > > >>>>>>>>>>>>>>>>> On Thu, Oct 29, 2020 at 10:21 AM Jingsong Li <
> >> > > >>>>>>>>>>>> jingsonglee0@gmail.com
> >> > > >>>>>>>>>>>>>>
> >> > > >>>>>>>>>>>>>>>>> wrote:
> >> > > >>>>>>>>>>>>>>>>>
> >> > > >>>>>>>>>>>>>>>>>> +1 to backport the FLIP-27 adjustments to
> >> > > >>> 1.11.x.
> >> > > >>>>>>>>>>>>>>>>>>
> >> > > >>>>>>>>>>>>>>>>>> If possible, that would be great. Many people
> >> > > >>> are
> >> > > >>>>>>> looking
> >> > > >>>>>>>>>>> forward
> >> > > >>>>>>>>>>>>> to
> >> > > >>>>>>>>>>>>>>> the
> >> > > >>>>>>>>>>>>>>>>>> FLIP-27 interface, but they don't want to take
> >> > > >>> the
> >> > > >>>>>> risk
> >> > > >>>>>>> to
> >> > > >>>>>>>>>>>> upgrade
> >> > > >>>>>>>>>>>>> to
> >> > > >>>>>>>>>>>>>>>>> 1.12
> >> > > >>>>>>>>>>>>>>>>>> (And wait 1.12). After all, 1.11 is a
> >> > > >> relatively
> >> > > >>>>>> stable
> >> > > >>>>>>>>>>> version.
> >> > > >>>>>>>>>>>>>>>>>>
> >> > > >>>>>>>>>>>>>>>>>> Best,
> >> > > >>>>>>>>>>>>>>>>>> Jingsong
> >> > > >>>>>>>>>>>>>>>>>>
> >> > > >>>>>>>>>>>>>>>>>> On Thu, Oct 29, 2020 at 1:24 AM Stephan Ewen <
> >> > > >>>>>>>>>> sewen@apache.org
> >> > > >>>>>>>>>>>>
> >> > > >>>>>>>>>>>>>>> wrote:
> >> > > >>>>>>>>>>>>>>>>>>
> >> > > >>>>>>>>>>>>>>>>>>> Thanks for starting this.
> >> > > >>>>>>>>>>>>>>>>>>>
> >> > > >>>>>>>>>>>>>>>>>>> +1 form my side to backport the FLIP-27
> >> > > >>>>> adjustments
> >> > > >>>>>> to
> >> > > >>>>>>>>>>> 1.11.x.
> >> > > >>>>>>>>>>>>>>>>>>>
> >> > > >>>>>>>>>>>>>>>>>>> There were quite a few changes, and I think
> >> > > >> we
> >> > > >>>>> need
> >> > > >>>>>> to
> >> > > >>>>>>>>>>>>> cherry-pick
> >> > > >>>>>>>>>>>>>>>>> them
> >> > > >>>>>>>>>>>>>>>>>> all
> >> > > >>>>>>>>>>>>>>>>>>> to not get some inconsistent mix of changes
> >> > > >>> and
> >> > > >>>>> many
> >> > > >>>>>>>>> merge
> >> > > >>>>>>>>>>>>>>> conflicts.
> >> > > >>>>>>>>>>>>>>>>>>> I made a list below of what we need to add
> >> > > >> to
> >> > > >>>>>>>>>> "release-1.11".
> >> > > >>>>>>>>>>>>>>>>>>>
> >> > > >>>>>>>>>>>>>>>>>>> *#### Core Source API Changes to backport
> >> > > >> (in
> >> > > >>>>>> REVERSE
> >> > > >>>>>>>>>> order)*
> >> > > >>>>>>>>>>>>>>>>>>>
> >> > > >>>>>>>>>>>>>>>>>>>  (Use: "git log
> >> > > >>>>>>>>>>>>>>>>>>>
> >> > > >>>>>>>>>>>>
> >> > > >>>>>
> flink-core/src/main/java/org/apache/flink/api/connector/source")
> >> > > >>>>>>>>>>>>>>>>>>>
> >> > > >>>>>>>>>>>>>>>>>>> commit
> >> > > >>> 162c072e9265a7b6dd9d6f5459eb7974091c4c4e
> >> > > >>>>>>>>>>>>>>>>>>> [FLINK-19492][core] Consolidate Source
> >> > > >> Events
> >> > > >>>>>> between
> >> > > >>>>>>>>>> Source
> >> > > >>>>>>>>>>>> API
> >> > > >>>>>>>>>>>>>>> and
> >> > > >>>>>>>>>>>>>>>>>> Split
> >> > > >>>>>>>>>>>>>>>>>>> Reader API
> >> > > >>>>>>>>>>>>>>>>>>>
> >> > > >>>>>>>>>>>>>>>>>>> commit
> >> > > >>> 428fbbad85379f71a9fbf1a1ab3e3dcfa9006436
> >> > > >>>>>>>>>>>>>>>>>>> [FLINK-19205][core] Add access to
> >> > > >>> configuration
> >> > > >>>>> and
> >> > > >>>>>>>>>> hostname
> >> > > >>>>>>>>>>> in
> >> > > >>>>>>>>>>>>> the
> >> > > >>>>>>>>>>>>>>>>>>> SourceReaderContext
> >> > > >>>>>>>>>>>>>>>>>>>
> >> > > >>>>>>>>>>>>>>>>>>> commit
> >> > > >>> 723e1790744ddc78d19c7c978442af1383f38d33
> >> > > >>>>>>>>>>>>>>>>>>> [hotfix][core] Add to Source Enumerator
> >> > > >>>>> convenience
> >> > > >>>>>>>>> methods
> >> > > >>>>>>>>>>> to
> >> > > >>>>>>>>>>>>>>> assign
> >> > > >>>>>>>>>>>>>>>>>>> single split
> >> > > >>>>>>>>>>>>>>>>>>>
> >> > > >>>>>>>>>>>>>>>>>>> *#### Connector Base Changes to Backport (in
> >> > > >>>>> REVERSE
> >> > > >>>>>>>>>> order)*
> >> > > >>>>>>>>>>>>>>>>>>>
> >> > > >>>>>>>>>>>>>>>>>>>  (Use: "git log
> >> > > >>>>>>> flink-connectors/flink-connector-base")
> >> > > >>>>>>>>>>>>>>>>>>>
> >> > > >>>>>>>>>>>>>>>>>>> commit
> >> > > >>> 401f56fe9d6b0271260edf9787cdcbfe4d03874d
> >> > > >>>>>>>>>>>>>>>>>>> [FLINK-19427][FLINK-19489][tests] Fix test
> >> > > >>>>>> conditions
> >> > > >>>>>>>>> for
> >> > > >>>>>>>>>>>>>>>>>>>
> >> > > >>>>>>> 'SplitFetcherTest.testNotifiesWhenGoingIdleConcurrent()'
> >> > > >>>>>>>>>>>>>>>>>>>
> >> > > >>>>>>>>>>>>>>>>>>> commit
> >> > > >>> 68c5c2ff779d82a1ff81ffaf60d8a1b283797db1
> >> > > >>>>>>>>>>>>>>>>>>> [FLINK-19448][connector base] Explicitly
> >> > > >> check
> >> > > >>>> for
> >> > > >>>>>>>>>>> un-expected
> >> > > >>>>>>>>>>>>>>>>> condition
> >> > > >>>>>>>>>>>>>>>>>>> that would leave an inconsistent state
> >> > > >>>>>>>>>>>>>>>>>>>
> >> > > >>>>>>>>>>>>>>>>>>> commit
> >> > > >>> 162c072e9265a7b6dd9d6f5459eb7974091c4c4e
> >> > > >>>>>>>>>>>>>>>>>>> [FLINK-19492][core] Consolidate Source
> >> > > >> Events
> >> > > >>>>>> between
> >> > > >>>>>>>>>> Source
> >> > > >>>>>>>>>>>> API
> >> > > >>>>>>>>>>>>>>> and
> >> > > >>>>>>>>>>>>>>>>>> Split
> >> > > >>>>>>>>>>>>>>>>>>> Reader API
> >> > > >>>>>>>>>>>>>>>>>>>
> >> > > >>>>>>>>>>>>>>>>>>> commit
> >> > > >>> c1ca7a4c7c21ec8868c14cdddd43c559625b794c
> >> > > >>>>>>>>>>>>>>>>>>> [refactor][tests] Move some source test
> >> > > >> utils
> >> > > >>>> from
> >> > > >>>>>>>>>>>>>>>>> flink-connector-base
> >> > > >>>>>>>>>>>>>>>>>> to
> >> > > >>>>>>>>>>>>>>>>>>> flink-core
> >> > > >>>>>>>>>>>>>>>>>>>
> >> > > >>>>>>>>>>>>>>>>>>> commit
> >> > > >>> ee5c4c211c35c70d28252363bbc8400453609977
> >> > > >>>>>>>>>>>>>>>>>>> [FLINK-19251][connectors] Avoid confusing
> >> > > >>> queue
> >> > > >>>>>>>>> handling in
> >> > > >>>>>>>>>>>>>>>>>>> "SplitReader.handleSplitsChanges()"
> >> > > >>>>>>>>>>>>>>>>>>>
> >> > > >>>>>>>>>>>>>>>>>>> commit
> >> > > >>> 5abef56b2bf85bcac786f6b16b6899b6cced7176
> >> > > >>>>>>>>>>>>>>>>>>> [FLINK-19250][connectors] Fix error
> >> > > >>> propagation
> >> > > >>>> in
> >> > > >>>>>>>>>> connector
> >> > > >>>>>>>>>>>> base
> >> > > >>>>>>>>>>>>>>>>>>> (SplitFetcherManager).
> >> > > >>>>>>>>>>>>>>>>>>>
> >> > > >>>>>>>>>>>>>>>>>>> commit
> >> > > >>> 8fcca837c55a9216595ee4c03038b52747098dbb
> >> > > >>>>>>>>>>>>>>>>>>> [hotfix][connectors] Improve JavaDocs for
> >> > > >>>>>>>>>>>>>>> SingleThreadFetcherManager
> >> > > >>>>>>>>>>>>>>>>>>>
> >> > > >>>>>>>>>>>>>>>>>>> commit
> >> > > >>> 4700bb5dde3303cbe98882f6beb7379425717b01
> >> > > >>>>>>>>>>>>>>>>>>> [FLINK-19225][connectors] Various small
> >> > > >>>>> improvements
> >> > > >>>>>>> to
> >> > > >>>>>>>>>>>>>>>>> SourceReaderBase
> >> > > >>>>>>>>>>>>>>>>>>> (part 2)
> >> > > >>>>>>>>>>>>>>>>>>>
> >> > > >>>>>>>>>>>>>>>>>>> commit
> >> > > >>> 12261c6b7ed6478a9b9f6a69cb58246b83cab9b7
> >> > > >>>>>>>>>>>>>>>>>>> [FLINK-17393][connectors] (follow-up) Wakeup
> >> > > >>> the
> >> > > >>>>>>>>>>> SplitFetchers
> >> > > >>>>>>>>>>>>> more
> >> > > >>>>>>>>>>>>>>>>>>> elegantly.
> >> > > >>>>>>>>>>>>>>>>>>>
> >> > > >>>>>>>>>>>>>>>>>>> commit
> >> > > >>> c60aaff0249bfd6b5871b7f82e03efc487a54d6b
> >> > > >>>>>>>>>>>>>>>>>>> [hotfix][tests] Extend test coverage for
> >> > > >>>>>>>>>>>>>>> FutureCompletingBlockingQueue
> >> > > >>>>>>>>>>>>>>>>>>>
> >> > > >>>>>>>>>>>>>>>>>>> commit
> >> > > >>> cef8a587d7fd2fe64cc644da5ed095d82e46f631
> >> > > >>>>>>>>>>>>>>>>>>> [FLINK-19245][connectors] Set default
> >> > > >> capacity
> >> > > >>>> for
> >> > > >>>>>>>>>>>>>>>>>>> FutureCompletingBlockingQueue.
> >> > > >>>>>>>>>>>>>>>>>>>
> >> > > >>>>>>>>>>>>>>>>>>> commit
> >> > > >>> 4ea95782b4c6a2538153d4d16ad3f4839c7de0fb
> >> > > >>>>>>>>>>>>>>>>>>> [FLINK-19223][connectors] Simplify
> >> > > >>> Availability
> >> > > >>>>>> Future
> >> > > >>>>>>>>>> Model
> >> > > >>>>>>>>>>> in
> >> > > >>>>>>>>>>>>>>> Base
> >> > > >>>>>>>>>>>>>>>>>>> Connector
> >> > > >>>>>>>>>>>>>>>>>>>
> >> > > >>>>>>>>>>>>>>>>>>> commit
> >> > > >>> 511857049ba30c8ff0ee56da551fa4a479dc583e
> >> > > >>>>>>>>>>>>>>>>>>> [FLINK-18128][connectors] Ensure idle split
> >> > > >>>>> fetchers
> >> > > >>>>>>>>> lead
> >> > > >>>>>>>>>> to
> >> > > >>>>>>>>>>>>>>>>> availability
> >> > > >>>>>>>>>>>>>>>>>>> notifications.
> >> > > >>>>>>>>>>>>>>>>>>>
> >> > > >>>>>>>>>>>>>>>>>>> commit
> >> > > >>> a8206467af0830dcb89623ea068b5ca3b3450c92
> >> > > >>>>>>>>>>>>>>>>>>> [refactor][core] Eagerly initialize the
> >> > > >>>> FetchTask
> >> > > >>>>> to
> >> > > >>>>>>>>>> support
> >> > > >>>>>>>>>>>>> proper
> >> > > >>>>>>>>>>>>>>>>> unit
> >> > > >>>>>>>>>>>>>>>>>>> testing
> >> > > >>>>>>>>>>>>>>>>>>>
> >> > > >>>>>>>>>>>>>>>>>>> commit
> >> > > >>> 3b2f54bcb437f98e6137c904045cc51072b5c06b
> >> > > >>>>>>>>>>>>>>>>>>> [hotfix][tests] Move constants in
> >> > > >>>> SplitFetcherTest
> >> > > >>>>>>>>> relevant
> >> > > >>>>>>>>>>> to
> >> > > >>>>>>>>>>>>> only
> >> > > >>>>>>>>>>>>>>>>> one
> >> > > >>>>>>>>>>>>>>>>>>> test into test method
> >> > > >>>>>>>>>>>>>>>>>>>
> >> > > >>>>>>>>>>>>>>>>>>> commit
> >> > > >>> d7625760a75a508bf05bcddc380bb4d62ee1743e
> >> > > >>>>>>>>>>>>>>>>>>> [FLINK-19225][connectors] Various small
> >> > > >>>>> improvements
> >> > > >>>>>>> to
> >> > > >>>>>>>>>>>>>>>>> SourceReaderBase
> >> > > >>>>>>>>>>>>>>>>>>>
> >> > > >>>>>>>>>>>>>>>>>>> commit
> >> > > >>> a5b0d3297748c1be47ad579a88f24df2255a8df1
> >> > > >>>>>>>>>>>>>>>>>>> [FLINK-17393][connectors] Wakeup the
> >> > > >>>> SplitFetchers
> >> > > >>>>>>> more
> >> > > >>>>>>>>>>>>> elegantly.
> >> > > >>>>>>>>>>>>>>>>>>>
> >> > > >>>>>>>>>>>>>>>>>>> commit
> >> > > >>> f42a3ebc3e81a034b7221a803c153636fef34903
> >> > > >>>>>>>>>>>>>>>>>>> [FLINK-18680][connectors] Make connector
> >> > > >> base
> >> > > >>>>>>>>>>>> RecordsWithSplitIds
> >> > > >>>>>>>>>>>>>>> more
> >> > > >>>>>>>>>>>>>>>>>>> lightweight.
> >> > > >>>>>>>>>>>>>>>>>>>
> >> > > >>>>>>>>>>>>>>>>>>> commit
> >> > > >>> e3d273de822b085183d09b275a445879ff94b350
> >> > > >>>>>>>>>>>>>>>>>>> [FLINK-19162][connectors] Add 'recycle()' to
> >> > > >>> the
> >> > > >>>>>>>>>>>>>>> RecordsWithSplitIds
> >> > > >>>>>>>>>>>>>>>>> to
> >> > > >>>>>>>>>>>>>>>>>>> support reuse of heavy objects.
> >> > > >>>>>>>>>>>>>>>>>>>
> >> > > >>>>>>>>>>>>>>>>>>> commit
> >> > > >>> 8ebc464c2520453a70001cd712abc8dee6ee89e0
> >> > > >>>>>>>>>>>>>>>>>>> [hotfix][testing] Add a set of
> >> > > >> parameterizable
> >> > > >>>>>> testing
> >> > > >>>>>>>>>> mocks
> >> > > >>>>>>>>>>>> for
> >> > > >>>>>>>>>>>>>>> the
> >> > > >>>>>>>>>>>>>>>>>> Split
> >> > > >>>>>>>>>>>>>>>>>>> Reader API
> >> > > >>>>>>>>>>>>>>>>>>>
> >> > > >>>>>>>>>>>>>>>>>>> commit
> >> > > >>> 930a07438be1185388d7150640f294dfe2a5d378
> >> > > >>>>>>>>>>>>>>>>>>> [hotfix][connectors] Add RequestSplitEvent
> >> > > >> to
> >> > > >>>>>>>>>>>>>>> 'flink-connector-base'
> >> > > >>>>>>>>>>>>>>>>>>>
> >> > > >>>>>>>>>>>>>>>>>>> commit
> >> > > >>> e945ce8a933bc378844782f784ca473c767ca159
> >> > > >>>>>>>>>>>>>>>>>>> [refactor][connectors] Backport of the
> >> > > >>>>>> connector-base
> >> > > >>>>>>>>>>> exception
> >> > > >>>>>>>>>>>>>>>>> handling
> >> > > >>>>>>>>>>>>>>>>>>> from the Kafka Connector Pull Request
> >> > > >>>>>>>>>>>>>>>>>>>
> >> > > >>>>>>>>>>>>>>>>>>>
> >> > > >>>>>>>>>>>>>>>>>>> On Wed, Oct 28, 2020 at 5:49 PM Steven Wu <
> >> > > >>>>>>>>>>>> stevenz3wu@gmail.com>
> >> > > >>>>>>>>>>>>>>>>> wrote:
> >> > > >>>>>>>>>>>>>>>>>>>
> >> > > >>>>>>>>>>>>>>>>>>>> I would love to see this FLIP-27 source
> >> > > >>>>> interface
> >> > > >>>>>>>>>>> improvement
> >> > > >>>>>>>>>>>>> [1]
> >> > > >>>>>>>>>>>>>>>>> made
> >> > > >>>>>>>>>>>>>>>>>> to
> >> > > >>>>>>>>>>>>>>>>>>>> 1.11.3.
> >> > > >>>>>>>>>>>>>>>>>>>>
> >> > > >>>>>>>>>>>>>>>>>>>> [1]
> >> > > >>>>>>> https://issues.apache.org/jira/browse/FLINK-19698
> >> > > >>>>>>>>>>>>>>>>>>>>
> >> > > >>>>>>>>>>>>>>>>>>>> On Wed, Oct 28, 2020 at 12:32 AM Tzu-Li
> >> > > >>>> (Gordon)
> >> > > >>>>>>> Tai <
> >> > > >>>>>>>>>>>>>>>>>>> tzulitai@apache.org>
> >> > > >>>>>>>>>>>>>>>>>>>> wrote:
> >> > > >>>>>>>>>>>>>>>>>>>>
> >> > > >>>>>>>>>>>>>>>>>>>>> Thanks for the replies so far!
> >> > > >>>>>>>>>>>>>>>>>>>>>
> >> > > >>>>>>>>>>>>>>>>>>>>> Just to provide a brief update on the
> >> > > >>> status
> >> > > >>>>> of
> >> > > >>>>>>>>>> blockers
> >> > > >>>>>>>>>>>> for
> >> > > >>>>>>>>>>>>>>>>> 1.11.3
> >> > > >>>>>>>>>>>>>>>>>> so
> >> > > >>>>>>>>>>>>>>>>>>>> far:
> >> > > >>>>>>>>>>>>>>>>>>>>>
> >> > > >>>>>>>>>>>>>>>>>>>>>
> >> > > >>>>>>>>>>>>>>>>>>>>> *PR opened, pending reviewer*-
> >> > > >>> [FLINK-19717]
> >> > > >>>>>>>>>>>>>>>>>> SourceReaderBase.pollNext
> >> > > >>>>>>>>>>>>>>>>>>>> may
> >> > > >>>>>>>>>>>>>>>>>>>>> return END_OF_INPUT if SplitReader.fetch
> >> > > >>>>> throws
> >> > > >>>>>> (
> >> > > >>>>>>>>>>>>>>>>>>>>>
> >> > > >>> https://github.com/apache/flink/pull/13776)
> >> > > >>>>>>>>>>>>>>>>>>>>>
> >> > > >>>>>>>>>>>>>>>>>>>>> *PR opened, reviewed + close to being
> >> > > >>>> merged*
> >> > > >>>>>>>>>>>>>>>>>>>>> - [FLINK-19741] Timer service should
> >> > > >> skip
> >> > > >>>>>>> restoring
> >> > > >>>>>>>>>> from
> >> > > >>>>>>>>>>>> raw
> >> > > >>>>>>>>>>>>>>> keyed
> >> > > >>>>>>>>>>>>>>>>>>> stream
> >> > > >>>>>>>>>>>>>>>>>>>>> if it isn't the writer (
> >> > > >>>>>>>>>>>>>>>>> https://github.com/apache/flink/pull/13761)
> >> > > >>>>>>>>>>>>>>>>>>>>> - [FLINK-19748] Raw keyed stream key
> >> > > >> group
> >> > > >>>>>>> iterator
> >> > > >>>>>>>>>>> should
> >> > > >>>>>>>>>>>> be
> >> > > >>>>>>>>>>>>>>>>>> skipping
> >> > > >>>>>>>>>>>>>>>>>>>>> unwritten key groups (
> >> > > >>>>>>>>>>>>>>> https://github.com/apache/flink/pull/13772)
> >> > > >>>>>>>>>>>>>>>>>>>>>
> >> > > >>>>>>>>>>>>>>>>>>>>> *Merged*
> >> > > >>>>>>>>>>>>>>>>>>>>> - [FLINK-19154] Application mode deletes
> >> > > >>> HA
> >> > > >>>>> data
> >> > > >>>>>>> in
> >> > > >>>>>>>>>> case
> >> > > >>>>>>>>>>> of
> >> > > >>>>>>>>>>>>>>>>> suspended
> >> > > >>>>>>>>>>>>>>>>>>>>> ZooKeeper connection
> >> > > >>>>>>>>>>>>>>>>>>>>> - [FLINK-19569] Upgrade ICU4J to 67.1+
> >> > > >>>>>>>>>>>>>>>>>>>>>
> >> > > >>>>>>>>>>>>>>>>>>>>> Right now as it seems, progress is
> >> > > >> mainly
> >> > > >>>>>> blocked
> >> > > >>>>>>>>> on a
> >> > > >>>>>>>>>>>>> reviewer
> >> > > >>>>>>>>>>>>>>>>> for
> >> > > >>>>>>>>>>>>>>>>>>>>> FLINK-19717.
> >> > > >>>>>>>>>>>>>>>>>>>>> Meanwhile, Xintong is keeping an eye on
> >> > > >>> test
> >> > > >>>>>>>>>>> instabilities
> >> > > >>>>>>>>>>>>> [1]
> >> > > >>>>>>>>>>>>>>> to
> >> > > >>>>>>>>>>>>>>>>> see
> >> > > >>>>>>>>>>>>>>>>>>> if
> >> > > >>>>>>>>>>>>>>>>>>>>> there are any fixes that should be
> >> > > >> applied
> >> > > >>>> to
> >> > > >>>>>>>>>>>> `release-1.11`.
> >> > > >>>>>>>>>>>>>>>>>>>>>
> >> > > >>>>>>>>>>>>>>>>>>>>> This is also a reminder, that if there
> >> > > >> are
> >> > > >>>>> other
> >> > > >>>>>>>>>> blockers
> >> > > >>>>>>>>>>>>> that
> >> > > >>>>>>>>>>>>>>> we
> >> > > >>>>>>>>>>>>>>>>>> need
> >> > > >>>>>>>>>>>>>>>>>>> to
> >> > > >>>>>>>>>>>>>>>>>>>>> be aware of, or a need to re-establish
> >> > > >>>>> estimated
> >> > > >>>>>>>>> time
> >> > > >>>>>>>>>> for
> >> > > >>>>>>>>>>>>>>> getting
> >> > > >>>>>>>>>>>>>>>>>> fixes
> >> > > >>>>>>>>>>>>>>>>>>>> in
> >> > > >>>>>>>>>>>>>>>>>>>>> and delay the RC for 1.11.3, please do
> >> > > >> let
> >> > > >>>> us
> >> > > >>>>>>> know!
> >> > > >>>>>>>>>>>>>>>>>>>>>
> >> > > >>>>>>>>>>>>>>>>>>>>> Cheers,
> >> > > >>>>>>>>>>>>>>>>>>>>> Gordon
> >> > > >>>>>>>>>>>>>>>>>>>>>
> >> > > >>>>>>>>>>>>>>>>>>>>> [1]
> >> > > >>>>>>>>>>>>>>>>>>>>>
> >> > > >>>>>>>>>>>>>>>>>>>>>
> >> > > >>>>>>>>>>>>>>>>>>>>
> >> > > >>>>>>>>>>>>>>>>>>>
> >> > > >>>>>>>>>>>>>>>>>>
> >> > > >>>>>>>>>>>>>>>>>
> >> > > >>>>>>>>>>>>>>>
> >> > > >>>>>>>>>>>>>
> >> > > >>>>>>>>>>>>
> >> > > >>>>>>>>>>>
> >> > > >>>>>>>>>>
> >> > > >>>>>>>>>
> >> > > >>>>>>>
> >> > > >>>>>>
> >> > > >>>>>
> >> > > >>>>
> >> > > >>>
> >> > > >>
> >> > >
> >> >
> >>
> https://issues.apache.org/jira/issues/?jql=project%20%3D%20FLINK%20AND%20status%20in%20(Open%2C%20%22In%20Progress%22%2C%20Reopened)%20AND%20affectedVersion%20in%20(1.11.0%2C%201.11.1%2C%201.11.2)%20AND%20labels%20%3D%20test-stability
> >> > > >>>>>>>>>>>>>>>>>>>>>
> >> > > >>>>>>>>>>>>>>>>>>>>> On Mon, Oct 26, 2020 at 9:43 PM Kostas
> >> > > >>>>> Kloudas <
> >> > > >>>>>>>>>>>>>>>>> kkloudas@apache.org>
> >> > > >>>>>>>>>>>>>>>>>>>>> wrote:
> >> > > >>>>>>>>>>>>>>>>>>>>>
> >> > > >>>>>>>>>>>>>>>>>>>>>> +1 for releasing Flink 1.11.3 as it
> >> > > >>>>> contains a
> >> > > >>>>>>>>> number
> >> > > >>>>>>>>>>> of
> >> > > >>>>>>>>>>>>>>>>> important
> >> > > >>>>>>>>>>>>>>>>>>>>>> fixes and thanks Gordon and Xintong
> >> > > >> for
> >> > > >>>>>>>>> volunteering.
> >> > > >>>>>>>>>>>>>>>>>>>>>>
> >> > > >>>>>>>>>>>>>>>>>>>>>> Cheers,
> >> > > >>>>>>>>>>>>>>>>>>>>>> Kostas
> >> > > >>>>>>>>>>>>>>>>>>>>>>
> >> > > >>>>>>>>>>>>>>>>>>>>>> On Mon, Oct 26, 2020 at 4:37 AM Yu Li
> >> > > >> <
> >> > > >>>>>>>>>>> carp84@gmail.com>
> >> > > >>>>>>>>>>>>>>> wrote:
> >> > > >>>>>>>>>>>>>>>>>>>>>>>
> >> > > >>>>>>>>>>>>>>>>>>>>>>> +1 for releasing Flink 1.11.3, and
> >> > > >>>> thanks
> >> > > >>>>>>> Gordon
> >> > > >>>>>>>>>> and
> >> > > >>>>>>>>>>>>>>> Xintong
> >> > > >>>>>>>>>>>>>>>>> for
> >> > > >>>>>>>>>>>>>>>>>>>>>>> volunteering as our release
> >> > > >> managers.
> >> > > >>>>>>>>>>>>>>>>>>>>>>>
> >> > > >>>>>>>>>>>>>>>>>>>>>>> Best Regards,
> >> > > >>>>>>>>>>>>>>>>>>>>>>> Yu
> >> > > >>>>>>>>>>>>>>>>>>>>>>>
> >> > > >>>>>>>>>>>>>>>>>>>>>>>
> >> > > >>>>>>>>>>>>>>>>>>>>>>> On Mon, 26 Oct 2020 at 09:45,
> >> > > >> Xintong
> >> > > >>>>> Song <
> >> > > >>>>>>>>>>>>>>>>>> tonysong820@gmail.com>
> >> > > >>>>>>>>>>>>>>>>>>>>>> wrote:
> >> > > >>>>>>>>>>>>>>>>>>>>>>>
> >> > > >>>>>>>>>>>>>>>>>>>>>>>> Thanks Gordan for starting this
> >> > > >>>>>> discussion.
> >> > > >>>>>>>>>>>>>>>>>>>>>>>> My pleasure to help with the
> >> > > >> release
> >> > > >>>>>>> process.
> >> > > >>>>>>>>>>>>>>>>>>>>>>>>
> >> > > >>>>>>>>>>>>>>>>>>>>>>>> Thank you~
> >> > > >>>>>>>>>>>>>>>>>>>>>>>>
> >> > > >>>>>>>>>>>>>>>>>>>>>>>> Xintong Song
> >> > > >>>>>>>>>>>>>>>>>>>>>>>>
> >> > > >>>>>>>>>>>>>>>>>>>>>>>>
> >> > > >>>>>>>>>>>>>>>>>>>>>>>>
> >> > > >>>>>>>>>>>>>>>>>>>>>>>> On Fri, Oct 23, 2020 at 11:29 PM
> >> > > >>> Till
> >> > > >>>>>>>>> Rohrmann <
> >> > > >>>>>>>>>>>>>>>>>>>> trohrmann@apache.org
> >> > > >>>>>>>>>>>>>>>>>>>>>>
> >> > > >>>>>>>>>>>>>>>>>>>>>>>> wrote:
> >> > > >>>>>>>>>>>>>>>>>>>>>>>>
> >> > > >>>>>>>>>>>>>>>>>>>>>>>>> Thanks for starting this
> >> > > >>> discussion
> >> > > >>>>>>> Gordon.
> >> > > >>>>>>>>>> There
> >> > > >>>>>>>>>>>> are
> >> > > >>>>>>>>>>>>>>> over
> >> > > >>>>>>>>>>>>>>>>>> 100
> >> > > >>>>>>>>>>>>>>>>>>>>> issues
> >> > > >>>>>>>>>>>>>>>>>>>>>>>>> which are fixed for 1.11.3.
> >> > > >> Hence
> >> > > >>> +1
> >> > > >>>>>> for a
> >> > > >>>>>>>>>>> soonish
> >> > > >>>>>>>>>>>>>>> 1.11.3
> >> > > >>>>>>>>>>>>>>>>>>>> release.
> >> > > >>>>>>>>>>>>>>>>>>>>>> Thanks
> >> > > >>>>>>>>>>>>>>>>>>>>>>>>> for volunteering as our release
> >> > > >>>>> managers
> >> > > >>>>>>>>> Gordon
> >> > > >>>>>>>>>>> and
> >> > > >>>>>>>>>>>>>>>>> Xintong!
> >> > > >>>>>>>>>>>>>>>>>>>>>>>>>
> >> > > >>>>>>>>>>>>>>>>>>>>>>>>> Cheers,
> >> > > >>>>>>>>>>>>>>>>>>>>>>>>> Till
> >> > > >>>>>>>>>>>>>>>>>>>>>>>>>
> >> > > >>>>>>>>>>>>>>>>>>>>>>>>> On Fri, Oct 23, 2020 at 5:02 PM
> >> > > >>>> Tzu-Li
> >> > > >>>>>>>>> (Gordon)
> >> > > >>>>>>>>>>>> Tai <
> >> > > >>>>>>>>>>>>>>>>>>>>>> tzulitai@apache.org
> >> > > >>>>>>>>>>>>>>>>>>>>>>>>>
> >> > > >>>>>>>>>>>>>>>>>>>>>>>>> wrote:
> >> > > >>>>>>>>>>>>>>>>>>>>>>>>>
> >> > > >>>>>>>>>>>>>>>>>>>>>>>>>> Hi,
> >> > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> >> > > >>>>>>>>>>>>>>>>>>>>>>>>>> Xintong and I would like to
> >> > > >>> start a
> >> > > >>>>>>>>> discussion
> >> > > >>>>>>>>>>> for
> >> > > >>>>>>>>>>>>>>>>> releasing
> >> > > >>>>>>>>>>>>>>>>>>>> Flink
> >> > > >>>>>>>>>>>>>>>>>>>>>>>> 1.11.3
> >> > > >>>>>>>>>>>>>>>>>>>>>>>>>> soon.
> >> > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> >> > > >>>>>>>>>>>>>>>>>>>>>>>>>> It seems like we already have a
> >> > > >>> few
> >> > > >>>>>>>>> pressing
> >> > > >>>>>>>>>>>> issues
> >> > > >>>>>>>>>>>>>>> that
> >> > > >>>>>>>>>>>>>>>>>> needs
> >> > > >>>>>>>>>>>>>>>>>>>> to
> >> > > >>>>>>>>>>>>>>>>>>>>> be
> >> > > >>>>>>>>>>>>>>>>>>>>>>>>>> included in a new hotfix
> >> > > >> release:
> >> > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> >> > > >>>>>>>>>>>>>>>>>>>>>>>>>>   - Heap-based timers’ restore
> >> > > >>>>>> behaviour
> >> > > >>>>>>>>> is
> >> > > >>>>>>>>>>>>> causing a
> >> > > >>>>>>>>>>>>>>>>>>> critical
> >> > > >>>>>>>>>>>>>>>>>>>>>> recovery
> >> > > >>>>>>>>>>>>>>>>>>>>>>>>>>   issue for StateFun [1] [2]
> >> > > >>> [3].
> >> > > >>>>>>>>>>>>>>>>>>>>>>>>>>   - There are several
> >> > > >> robustness
> >> > > >>>>>> issues
> >> > > >>>>>>>>> for
> >> > > >>>>>>>>>> the
> >> > > >>>>>>>>>>>>>>> FLIP-27
> >> > > >>>>>>>>>>>>>>>>> new
> >> > > >>>>>>>>>>>>>>>>>>>>> source
> >> > > >>>>>>>>>>>>>>>>>>>>>> API,
> >> > > >>>>>>>>>>>>>>>>>>>>>>>>>>   such as [4]. We already have
> >> > > >>>> some
> >> > > >>>>>>> users
> >> > > >>>>>>>>>> using
> >> > > >>>>>>>>>>>> the
> >> > > >>>>>>>>>>>>>>>>> FLIP-27
> >> > > >>>>>>>>>>>>>>>>>>> API
> >> > > >>>>>>>>>>>>>>>>>>>>>> with
> >> > > >>>>>>>>>>>>>>>>>>>>>>>>>> 1.11.x,
> >> > > >>>>>>>>>>>>>>>>>>>>>>>>>>   so it would be important to
> >> > > >>> get
> >> > > >>>>>> those
> >> > > >>>>>>>>> fixes
> >> > > >>>>>>>>>>> in
> >> > > >>>>>>>>>>>>> for
> >> > > >>>>>>>>>>>>>>>>> 1.11.x
> >> > > >>>>>>>>>>>>>>>>>>> as
> >> > > >>>>>>>>>>>>>>>>>>>>>> well.
> >> > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> >> > > >>>>>>>>>>>>>>>>>>>>>>>>>> Apart from the issues that are
> >> > > >>>>> already
> >> > > >>>>>>>>> marked
> >> > > >>>>>>>>>> as
> >> > > >>>>>>>>>>>>>>> blocker
> >> > > >>>>>>>>>>>>>>>>> for
> >> > > >>>>>>>>>>>>>>>>>>>>> 1.11.3
> >> > > >>>>>>>>>>>>>>>>>>>>>> in
> >> > > >>>>>>>>>>>>>>>>>>>>>>>> our
> >> > > >>>>>>>>>>>>>>>>>>>>>>>>>> JIRA [5], please let us know in
> >> > > >>>> this
> >> > > >>>>>>>>> thread if
> >> > > >>>>>>>>>>>> there
> >> > > >>>>>>>>>>>>>>> is
> >> > > >>>>>>>>>>>>>>>>>>> already
> >> > > >>>>>>>>>>>>>>>>>>>>>> ongoing
> >> > > >>>>>>>>>>>>>>>>>>>>>>>>>> work for other important fixes
> >> > > >>> that
> >> > > >>>>> we
> >> > > >>>>>>>>> should
> >> > > >>>>>>>>>>> try
> >> > > >>>>>>>>>>>> to
> >> > > >>>>>>>>>>>>>>>>>> include.
> >> > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> >> > > >>>>>>>>>>>>>>>>>>>>>>>>>> Xintong and I would like to
> >> > > >>>> volunteer
> >> > > >>>>>> for
> >> > > >>>>>>>>>>> managing
> >> > > >>>>>>>>>>>>>>> this
> >> > > >>>>>>>>>>>>>>>>>>> release,
> >> > > >>>>>>>>>>>>>>>>>>>>> and
> >> > > >>>>>>>>>>>>>>>>>>>>>>>> will
> >> > > >>>>>>>>>>>>>>>>>>>>>>>>>> try to communicate the priority
> >> > > >>> of
> >> > > >>>>>>> pending
> >> > > >>>>>>>>>>>> blockers
> >> > > >>>>>>>>>>>>>>> over
> >> > > >>>>>>>>>>>>>>>>> the
> >> > > >>>>>>>>>>>>>>>>>>>> next
> >> > > >>>>>>>>>>>>>>>>>>>>>> few
> >> > > >>>>>>>>>>>>>>>>>>>>>>>>>> days.
> >> > > >>>>>>>>>>>>>>>>>>>>>>>>>> Since the aforementioned issues
> >> > > >>> are
> >> > > >>>>>> quite
> >> > > >>>>>>>>>>>> critical,
> >> > > >>>>>>>>>>>>>>> we’d
> >> > > >>>>>>>>>>>>>>>>>> like
> >> > > >>>>>>>>>>>>>>>>>>> to
> >> > > >>>>>>>>>>>>>>>>>>>>> aim
> >> > > >>>>>>>>>>>>>>>>>>>>>>>>>> for a *feature
> >> > > >>>>>>>>>>>>>>>>>>>>>>>>>> freeze by the end of next week
> >> > > >>>> (Oct.
> >> > > >>>>>>> 30th)*
> >> > > >>>>>>>>>> and
> >> > > >>>>>>>>>>>>> start
> >> > > >>>>>>>>>>>>>>> the
> >> > > >>>>>>>>>>>>>>>>>>>> release
> >> > > >>>>>>>>>>>>>>>>>>>>>> voting
> >> > > >>>>>>>>>>>>>>>>>>>>>>>>>> process the week after.
> >> > > >>>>>>>>>>>>>>>>>>>>>>>>>> If that is too short of a
> >> > > >> notice
> >> > > >>>> and
> >> > > >>>>>> you
> >> > > >>>>>>>>> might
> >> > > >>>>>>>>>>>> need
> >> > > >>>>>>>>>>>>>>> more
> >> > > >>>>>>>>>>>>>>>>>> time,
> >> > > >>>>>>>>>>>>>>>>>>>>>> please
> >> > > >>>>>>>>>>>>>>>>>>>>>>>> let
> >> > > >>>>>>>>>>>>>>>>>>>>>>>>>> us know!
> >> > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> >> > > >>>>>>>>>>>>>>>>>>>>>>>>>> Cheers,
> >> > > >>>>>>>>>>>>>>>>>>>>>>>>>> Gordon
> >> > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> >> > > >>>>>>>>>>>>>>>>>>>>>>>>>> [1]
> >> > > >>>>>>>>>>>>> https://issues.apache.org/jira/browse/FLINK-19692
> >> > > >>>>>>>>>>>>>>>>>>>>>>>>>> [2]
> >> > > >>>>>>>>>>>>> https://issues.apache.org/jira/browse/FLINK-19741
> >> > > >>>>>>>>>>>>>>>>>>>>>>>>>> [3]
> >> > > >>>>>>>>>>>>> https://issues.apache.org/jira/browse/FLINK-19748
> >> > > >>>>>>>>>>>>>>>>>>>>>>>>>> [4]
> >> > > >>>>>>>>>>>>> https://issues.apache.org/jira/browse/FLINK-19717
> >> > > >>>>>>>>>>>>>>>>>>>>>>>>>> [5]
> >> > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> >> > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> >> > > >>>>>>>>>>>>>>>>>>>>>>>>
> >> > > >>>>>>>>>>>>>>>>>>>>>>
> >> > > >>>>>>>>>>>>>>>>>>>>>
> >> > > >>>>>>>>>>>>>>>>>>>>
> >> > > >>>>>>>>>>>>>>>>>>>
> >> > > >>>>>>>>>>>>>>>>>>
> >> > > >>>>>>>>>>>>>>>>>
> >> > > >>>>>>>>>>>>>>>
> >> > > >>>>>>>>>>>>>
> >> > > >>>>>>>>>>>>
> >> > > >>>>>>>>>>>
> >> > > >>>>>>>>>>
> >> > > >>>>>>>>>
> >> > > >>>>>>>
> >> > > >>>>>>
> >> > > >>>>>
> >> > > >>>>
> >> > > >>>
> >> > > >>
> >> > >
> >> >
> >>
> https://issues.apache.org/jira/issues/?jql=project%20%3D%20FLINK%20AND%20priority%20%3D%20Blocker%20AND%20fixVersion%20%3D%201.11.3
> >> > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> >> > > >>>>>>>>>>>>>>>>>>>>>>>>>
> >> > > >>>>>>>>>>>>>>>>>>>>>>>>
> >> > > >>>>>>>>>>>>>>>>>>>>>>
> >> > > >>>>>>>>>>>>>>>>>>>>>
> >> > > >>>>>>>>>>>>>>>>>>>>
> >> > > >>>>>>>>>>>>>>>>>>>
> >> > > >>>>>>>>>>>>>>>>>>
> >> > > >>>>>>>>>>>>>>>>>>
> >> > > >>>>>>>>>>>>>>>>>> --
> >> > > >>>>>>>>>>>>>>>>>> Best, Jingsong Lee
> >> > > >>>>>>>>>>>>>>>>>>
> >> > > >>>>>>>>>>>>>>>>>
> >> > > >>>>>>>>>>>>>>>>
> >> > > >>>>>>>>>>>>>>>
> >> > > >>>>>>>>>>>>>>
> >> > > >>>>>>>>>>>>>
> >> > > >>>>>>>>>>>>
> >> > > >>>>>>>>>>>
> >> > > >>>>>>>>>>
> >> > > >>>>>>>>>
> >> > > >>>>>>>>
> >> > > >>>>>>>
> >> > > >>>>>>
> >> > > >>>>>
> >> > > >>>>
> >> > > >>>
> >> > > >>
> >> > >
> >> > >
> >> > >
> >> >
> >>
> >
>

Re: [DISCUSS] Releasing Apache Flink 1.11.3

Posted by Xintong Song <to...@gmail.com>.
Hi devs,

Updates on the progress of preparing the 1.11.3 release.

We are approaching the creation of our first release candidate. Thanks all
for the efforts so far.

# Ongoing threads

   - *FLIP-27 backportings:* 74 out of 102 commits are already backported.
   Stephan and Becket are actively working on this.
   - *Blockers:* There are 2 remaining blockers, both have been fixed on
   the master branch and should be backported along with the FLIP-27 changes.

As soon as the above threads are done, we will create our first release
candidate. If there's any other issues that you believe should be a release
blocker, please reach out to either me or Gordon.

# JIRA version

Version 1.11.4 has been created on JIRA. I've already tagged some of the
tickets with the new fix version. Please feel free to continue with the
efforts and merge once they are ready. I'll double check and update the fix
versions before the release.

Thank you~

Xintong Song



On Thu, Nov 12, 2020 at 1:31 PM Xintong Song <to...@gmail.com> wrote:

> Thanks for the notice and fix, Roman.
>
> Thank you~
>
> Xintong Song
>
>
>
> On Wed, Nov 11, 2020 at 5:53 PM Khachatryan Roman <
> khachatryan.roman@gmail.com> wrote:
>
>> Hi,
>>
>> I'd like FLINK-20079 [1] to be merged into 1.11 and included in 1.11.3.
>>
>> [1] https://issues.apache.org/jira/browse/FLINK-20079
>>
>> Regards,
>> Roman
>>
>>
>> On Tue, Nov 10, 2020 at 8:21 AM Xintong Song <to...@gmail.com>
>> wrote:
>>
>> > Thanks for the notice, Dian.
>> >
>> > Thank you~
>> >
>> > Xintong Song
>> >
>> >
>> >
>> > On Tue, Nov 10, 2020 at 10:19 AM Dian Fu <di...@gmail.com> wrote:
>> >
>> > > Hi Xintong,
>> > >
>> > > I want to bring one more issue to your attention [1]. The test case
>> > > UnalignedCheckpointCompatibilityITCase.test failed several times in
>> the
>> > > last nightly test of release-1.11. We need to figure out if it's just
>> an
>> > > instable test or caused by recent changes.
>> > >
>> > > [1] https://issues.apache.org/jira/browse/FLINK-20065
>> > >
>> > > > 在 2020年11月10日，上午9:24，Xintong Song <to...@gmail.com> 写道：
>> > > >
>> > > > Thanks for the replies.
>> > > >
>> > > > Thank you~
>> > > >
>> > > > Xintong Song
>> > > >
>> > > >
>> > > >
>> > > > On Tue, Nov 10, 2020 at 1:09 AM Becket Qin <be...@gmail.com>
>> > wrote:
>> > > >
>> > > >> Hi Xintong,
>> > > >>
>> > > >> Thanks for driving the release. Just want to sync up on the FLIP-27
>> > > >> backporting. Stephan and I are still trying to backport a bunch of
>> > > patches
>> > > >> of Source to 1.11.3. Including:
>> > > >>
>> > > >> [FLINK-19698][connector/common] Add a close() method to the
>> > SplitReader.
>> > > >> [FLINK-19717] SourceReaderBase.pollNext may return END_OF_INPUT if
>> > > >> SplitReader.fetch throws
>> > > >> [FLINK-19535] [connector/common] Avoid failing a job multiple
>> times in
>> > > >> SourceCoordinator.
>> > > >> [FLINK-19265] [FLINK-20049][core] Source API final adjustments.
>> > > >>
>> > > >> and a few more fixes.
>> > > >>
>> > > >> We are currently trying to fix them in 1.12 first so it might take
>> a
>> > > little
>> > > >> longer to backport them to 1.11.3. I think it will probably take
>> us a
>> > > few
>> > > >> more days to finish the backport. So that would roughly be the end
>> of
>> > > this
>> > > >> week.
>> > > >>
>> > > >> Thanks,
>> > > >>
>> > > >> Jiangjie (Becket) Qin
>> > > >>
>> > > >>
>> > > >>
>> > > >>
>> > > >> On Mon, Nov 9, 2020 at 9:57 PM Till Rohrmann <trohrmann@apache.org
>> >
>> > > wrote:
>> > > >>
>> > > >>> Yes, I've downgraded FLINK-19816 to critical.
>> > > >>>
>> > > >>> Cheers,
>> > > >>> Till
>> > > >>>
>> > > >>> On Mon, Nov 9, 2020 at 10:19 AM Xintong Song <
>> tonysong820@gmail.com>
>> > > >>> wrote:
>> > > >>>
>> > > >>>> Thanks for the notice, Till.
>> > > >>>>
>> > > >>>> I just checked and found FLINK-20033 is already fixed. Shall we
>> also
>> > > >>>> downgrade FLINK-19816 to `Critical`?
>> > > >>>>
>> > > >>>> Thank you~
>> > > >>>>
>> > > >>>> Xintong Song
>> > > >>>>
>> > > >>>>
>> > > >>>>
>> > > >>>> On Mon, Nov 9, 2020 at 4:42 PM Till Rohrmann <
>> trohrmann@apache.org>
>> > > >>> wrote:
>> > > >>>>
>> > > >>>>> I would like to bring one more critical issue to your attention
>> > which
>> > > >>> is
>> > > >>>>> FLINK-20033 [1]. I believe that this issue is actually causing
>> what
>> > > >> has
>> > > >>>>> been reported in FLINK-19816 [2]. I hope to have it fixed by the
>> > end
>> > > >> of
>> > > >>>>> today. Once FLINK-20033 is fixed, I think that we don't have to
>> > block
>> > > >>> the
>> > > >>>>> release on FLINK-19816.
>> > > >>>>>
>> > > >>>>> [1] https://issues.apache.org/jira/browse/FLINK-20033
>> > > >>>>> [2] https://issues.apache.org/jira/browse/FLINK-19816
>> > > >>>>>
>> > > >>>>> Cheers,
>> > > >>>>> Till
>> > > >>>>>
>> > > >>>>> On Mon, Nov 9, 2020 at 4:05 AM Xintong Song <
>> tonysong820@gmail.com
>> > >
>> > > >>>> wrote:
>> > > >>>>>
>> > > >>>>>> Hi devs,
>> > > >>>>>>
>> > > >>>>>> I'd like to provide an update on the progress of preparing
>> release
>> > > >>>>> 1.11.3.
>> > > >>>>>>
>> > > >>>>>> *Blockers*
>> > > >>>>>> We currently have 3 remaining blockers. (3 resolved and 1
>> emerged
>> > > >>>>> compared
>> > > >>>>>> to last week)
>> > > >>>>>>
>> > > >>>>>> - [FLINK-19698] Add close() method and onCheckpointComplete()
>> to
>> > > >> the
>> > > >>>>>> Source.
>> > > >>>>>> The issue has been fixed on the master branch. It's currently
>> > > >> blocked
>> > > >>>> on
>> > > >>>>>> the FLIP-27 backportings to backport it to the 1.11 branch.
>> > > >>>>>>
>> > > >>>>>> - [FLINK-19717] SourceReaderBase.pollNext may return
>> END_OF_INPUT
>> > > >> if
>> > > >>>>>> SplitReader.fetch throws
>> > > >>>>>> A PR has been opened and reviewed. From the discussions on the
>> PR,
>> > > >> it
>> > > >>>>> looks
>> > > >>>>>> close to mergeable.
>> > > >>>>>>
>> > > >>>>>> - [FLINK-19816] Flink restored from a wrong checkpoint (a very
>> old
>> > > >>> one
>> > > >>>>> and
>> > > >>>>>> not the last completed one)
>> > > >>>>>> This is a newly emerged blocker and Matthias is working on it.
>> > > >>>>>>
>> > > >>>>>> *Test Instabilities*
>> > > >>>>>> We currently have 27 test instabilities[1].
>> > > >>>>>> AFAIK, none of them are as serious as to block the 1.11.3
>> release.
>> > > >>>>>>
>> > > >>>>>> *FLIP-27 Backprotings*
>> > > >>>>>>
>> > > >>>>>> I noticed that there's no jira issues opened on the FLIP-27
>> > > >>> backporting
>> > > >>>>>> efforts, which is part of the major efforts planned for the
>> 1.11.3
>> > > >>>>> release,
>> > > >>>>>> making it hard to track the progress.
>> > > >>>>>>
>> > > >>>>>>
>> > > >>>>>> @Stephan and @Becket, could you please share the updates on the
>> > > >>>>> backporting
>> > > >>>>>> efforts? How is the progress and when are the efforts expected
>> to
>> > > >> be
>> > > >>>>>> finished? It would be appreciated and helpful if we can have a
>> > jira
>> > > >>>>> ticket
>> > > >>>>>> to track the backporing progress publicly.
>> > > >>>>>>
>> > > >>>>>>
>> > > >>>>>> Thank you~
>> > > >>>>>>
>> > > >>>>>> Xintong Song
>> > > >>>>>>
>> > > >>>>>>
>> > > >>>>>> [1]
>> > > >>>>>>
>> > > >>>>>>
>> > > >>>>>
>> > > >>>>
>> > > >>>
>> > > >>
>> > >
>> >
>> https://issues.apache.org/jira/browse/FLINK-16908?filter=12348580&jql=project%20%3D%20FLINK%20AND%20resolution%20%3D%20Unresolved%20AND%20affectedVersion%20in%20(1.11.0%2C%201.11.1%2C%201.11.2%2C%201.11.3)%20AND%20labels%20%3D%20test-stability%20ORDER%20BY%20updated%20DESC%2C%20created%20DESC
>> > > >>>>>>
>> > > >>>>>> On Wed, Nov 4, 2020 at 3:58 PM Tzu-Li (Gordon) Tai <
>> > > >>>> tzulitai@apache.org>
>> > > >>>>>> wrote:
>> > > >>>>>>
>> > > >>>>>>> Thanks for the efforts so far with pushing for 1.11.3.
>> > > >>>>>>>
>> > > >>>>>>> I'd like to provide a mid-week update on how we're looking
>> with
>> > > >> all
>> > > >>>> the
>> > > >>>>>>> pending blockers and backports:
>> > > >>>>>>>
>> > > >>>>>>> *Blockers**:*
>> > > >>>>>>>
>> > > >>>>>>> Currently all blockers either have an open PR, or have been
>> > > >> merged.
>> > > >>>>> I'll
>> > > >>>>>>> highlight below the blockers *that still require reviewing
>> > > >> efforts
>> > > >>>> for
>> > > >>>>>> them
>> > > >>>>>>> to move forward:*
>> > > >>>>>>>
>> > > >>>>>>> - [FLINK-19909] Flink application in attach mode could not
>> > > >>> terminate
>> > > >>>>> when
>> > > >>>>>>> the only job is canceled. PR:
>> > > >>>>> https://github.com/apache/flink/pull/13911
>> > > >>>>>>> - [FLINK-19717] SourceReaderBase.pollNext may return
>> END_OF_INPUT
>> > > >>> if
>> > > >>>>>>> SplitReader.fetch throws. PR:
>> > > >>>>> https://github.com/apache/flink/pull/13776
>> > > >>>>>>>
>> > > >>>>>>> The above PRs currently have no reviews at all yet. They seem
>> to
>> > > >>>>> already
>> > > >>>>>>> have designated reviewers.
>> > > >>>>>>>
>> > > >>>>>>>
>> > > >>>>>>> *Backports:*
>> > > >>>>>>> - FLIP-27 robustness improvement backports: Stephan is
>> currently
>> > > >>>>> working
>> > > >>>>>> on
>> > > >>>>>>> backporting several FLIP-27 changes. There are no PRs yet for
>> the
>> > > >>>>>>> backported changes.
>> > > >>>>>>> - DataStreamUtils.collect() refactorings backport: Steven Wu
>> > > >>>> mentioned
>> > > >>>>> to
>> > > >>>>>>> backport this, but AFAIK this isn't assigned to anyone yet.
>> > > >>>>>>>
>> > > >>>>>>> *ETAs*:
>> > > >>>>>>>
>> > > >>>>>>> I'd like to request ETAs for the remaining backports, to
>> prevent
>> > > >> a
>> > > >>>>> creep
>> > > >>>>>> in
>> > > >>>>>>> the scope of this bugfix release.
>> > > >>>>>>>
>> > > >>>>>>> We already have Flink users that would benefit from fixes that
>> > > >> have
>> > > >>>>> been
>> > > >>>>>>> merged to release-1.11 already, so technically speaking the
>> > > >>> backports
>> > > >>>>>>> should be considered "nice-to-have" (to the best of my
>> knowledge
>> > > >> of
>> > > >>>> the
>> > > >>>>>>> changes) and could potentially be moved to a follow-up 1.11.4.
>> > > >>>>>>> Most notably, the Stateful Functions project is already
>> waiting
>> > > >> on
>> > > >>>>> Flink
>> > > >>>>>>> 1.11.3 to address critical recovery issues (please see the
>> > > >> StateFun
>> > > >>>>> 2.2.1
>> > > >>>>>>> release discussion thread [1]).
>> > > >>>>>>>
>> > > >>>>>>> @Stephan Ewen <se...@apache.org> @Becket Qin <
>> > > >> becket.qin@gmail.com
>> > > >>>>
>> > > >>>>>> could
>> > > >>>>>>> you provide an ETA for the FLIP-27 backports? It would help to
>> > > >> get
>> > > >>> a
>> > > >>>>>> better
>> > > >>>>>>> estimate to decide how we proceed here.
>> > > >>>>>>>
>> > > >>>>>>> Cheers,
>> > > >>>>>>> Gordon
>> > > >>>>>>>
>> > > >>>>>>> [1]
>> > > >>>>>>>
>> > > >>>>>>>
>> > > >>>>>>
>> > > >>>>>
>> > > >>>>
>> > > >>>
>> > > >>
>> > >
>> >
>> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Releasing-StateFun-hotfix-version-2-2-1-td46239.html
>> > > >>>>>>>
>> > > >>>>>>> On Wed, Nov 4, 2020 at 3:16 PM Tzu-Li (Gordon) Tai <
>> > > >>>>> tzulitai@apache.org>
>> > > >>>>>>> wrote:
>> > > >>>>>>>
>> > > >>>>>>>>> The collect() utils can be picked back, I see no issue with
>> > > >>> that
>> > > >>>>> (it
>> > > >>>>>> is
>> > > >>>>>>>> isolated utilities).
>> > > >>>>>>>>
>> > > >>>>>>>> Just checking on all the requested backports mentioned in
>> this
>> > > >>>>> thread,
>> > > >>>>>>> and
>> > > >>>>>>>> figuring out which ones seem to still be unassigned / open.
>> > > >>>>>>>>
>> > > >>>>>>>> Is someone working on backporting
>> > > >>>>>>>>
>> > > >>>>>>>
>> > > >>>>>>
>> > > >>>>>
>> > > >>>>
>> > > >>>
>> > > >>
>> > >
>> >
>> https://github.com/apache/flink/commit/09a7a66b7313fea64817fe960a8da1265b428efc
>> > > >>>>>>>> to release-1.11 at the moment?
>> > > >>>>>>>>
>> > > >>>>>>>> On Wed, Nov 4, 2020 at 1:39 AM Steven Wu <
>> stevenz3wu@gmail.com
>> > > >>>
>> > > >>>>> wrote:
>> > > >>>>>>>>
>> > > >>>>>>>>> @Stephan Ewen <se...@apache.org> yeah, we can do that.
>> don't
>> > > >>>> worry
>> > > >>>>>>> about
>> > > >>>>>>>>> it. your earlier email had the perfect explanation on why
>> file
>> > > >>>>> source
>> > > >>>>>>>>> shouldn't be backported.
>> > > >>>>>>>>>
>> > > >>>>>>>>> On Tue, Nov 3, 2020 at 3:37 AM Stephan Ewen <
>> sewen@apache.org
>> > > >>>
>> > > >>>>> wrote:
>> > > >>>>>>>>>
>> > > >>>>>>>>>> @Steven would it be possible to initially copy some of the
>> > > >>> code
>> > > >>>>> into
>> > > >>>>>>> the
>> > > >>>>>>>>>> iceberg source and later replace it by a dependency on the
>> > > >>> Flink
>> > > >>>>>> file
>> > > >>>>>>>>>> source?
>> > > >>>>>>>>>>
>> > > >>>>>>>>>> On Mon, Nov 2, 2020 at 8:33 PM Steven Wu <
>> > > >>> stevenz3wu@gmail.com>
>> > > >>>>>>> wrote:
>> > > >>>>>>>>>>
>> > > >>>>>>>>>>> Stephan, thanks a lot for explaining the file connector.
>> > > >>> that
>> > > >>>>>> makes
>> > > >>>>>>>>>> sense.
>> > > >>>>>>>>>>>
>> > > >>>>>>>>>>> I was asking because we were trying to reuse some of the
>> > > >>>>>>>>> implementations
>> > > >>>>>>>>>> in
>> > > >>>>>>>>>>> the file source for Iceberg source. Flink Iceberg source
>> > > >>> lives
>> > > >>>>> in
>> > > >>>>>>> the
>> > > >>>>>>>>>>> Iceberg repo, which is not possible to code against the
>> > > >>> master
>> > > >>>>>>> branch
>> > > >>>>>>>>> of
>> > > >>>>>>>>>>> the Flink code.
>> > > >>>>>>>>>>>
>> > > >>>>>>>>>>> On Mon, Nov 2, 2020 at 3:31 AM Stephan Ewen <
>> > > >>> sewen@apache.org
>> > > >>>>>
>> > > >>>>>>> wrote:
>> > > >>>>>>>>>>>
>> > > >>>>>>>>>>>> Hi Steven!
>> > > >>>>>>>>>>>>
>> > > >>>>>>>>>>>> So far there are no plans to pick back the file system
>> > > >>>>> connector
>> > > >>>>>>>>> code.
>> > > >>>>>>>>>>> This
>> > > >>>>>>>>>>>> is still evolving and not finalized for 1.12, so I don't
>> > > >>>> feel
>> > > >>>>> it
>> > > >>>>>>> is
>> > > >>>>>>>>> a
>> > > >>>>>>>>>>> good
>> > > >>>>>>>>>>>> candidate to be backported.
>> > > >>>>>>>>>>>> However, with the base connector changes backported, you
>> > > >>>>> should
>> > > >>>>>> be
>> > > >>>>>>>>> able
>> > > >>>>>>>>>>> to
>> > > >>>>>>>>>>>> run the file connector code from master against 1.11.3.
>> > > >>>>>>>>>>>>
>> > > >>>>>>>>>>>> The collect() utils can be picked back, I see no issue
>> > > >>> with
>> > > >>>>> that
>> > > >>>>>>>>> (it is
>> > > >>>>>>>>>>>> isolated utilities).
>> > > >>>>>>>>>>>>
>> > > >>>>>>>>>>>> Best,
>> > > >>>>>>>>>>>> Stephan
>> > > >>>>>>>>>>>>
>> > > >>>>>>>>>>>>
>> > > >>>>>>>>>>>> On Mon, Nov 2, 2020 at 3:02 AM Steven Wu <
>> > > >>>>> stevenz3wu@gmail.com>
>> > > >>>>>>>>> wrote:
>> > > >>>>>>>>>>>>
>> > > >>>>>>>>>>>>> Basically, it would be great to get the latest code in
>> > > >>> the
>> > > >>>>>>>>>>>>> flink-connector-files (FLIP-27).
>> > > >>>>>>>>>>>>>
>> > > >>>>>>>>>>>>> On Sat, Oct 31, 2020 at 9:57 AM Steven Wu <
>> > > >>>>>> stevenz3wu@gmail.com
>> > > >>>>>>>>
>> > > >>>>>>>>>>> wrote:
>> > > >>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>> Stephan, it will be great if we can also backport
>> > > >> the
>> > > >>>>>>>>>> DataStreamUtils
>> > > >>>>>>>>>>>>>> related commits that help with collecting output
>> > > >> from
>> > > >>>>>>> unbounded
>> > > >>>>>>>>>>>> streams.
>> > > >>>>>>>>>>>>>> e.g.
>> > > >>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>
>> > > >>>>>>>>>>>>
>> > > >>>>>>>>>>>
>> > > >>>>>>>>>>
>> > > >>>>>>>>>
>> > > >>>>>>>
>> > > >>>>>>
>> > > >>>>>
>> > > >>>>
>> > > >>>
>> > > >>
>> > >
>> >
>> https://github.com/apache/flink/commit/09a7a66b7313fea64817fe960a8da1265b428efc
>> > > >>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>> I tried to copy and paste the code to unblock
>> > > >> myself.
>> > > >>>> but
>> > > >>>>> it
>> > > >>>>>>>>>> quickly
>> > > >>>>>>>>>>>> got
>> > > >>>>>>>>>>>>>> into the rabbit hole of more and more code.
>> > > >>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>> On Fri, Oct 30, 2020 at 11:02 AM Stephan Ewen <
>> > > >>>>>>> sewen@apache.org
>> > > >>>>>>>>>>
>> > > >>>>>>>>>>>> wrote:
>> > > >>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>> I have started with backporting the source API
>> > > >>> changes.
>> > > >>>>>> Some
>> > > >>>>>>>>> minor
>> > > >>>>>>>>>>>>>>> conflicts to solve, will need a bit more to finish
>> > > >>>> this.
>> > > >>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>> On Fri, Oct 30, 2020 at 7:25 AM Tzu-Li (Gordon)
>> > > >> Tai <
>> > > >>>>>>>>>>>>> tzulitai@apache.org>
>> > > >>>>>>>>>>>>>>> wrote:
>> > > >>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>> @Stephan Ewen <se...@apache.org>
>> > > >>>>>>>>>>>>>>>> Are there already plans or ongoing efforts for
>> > > >>>>>> backporting
>> > > >>>>>>>>> the
>> > > >>>>>>>>>>> list
>> > > >>>>>>>>>>>> of
>> > > >>>>>>>>>>>>>>>> FLIP-27 changes that you posted?
>> > > >>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>> On Thu, Oct 29, 2020 at 7:08 PM Xintong Song <
>> > > >>>>>>>>>>> tonysong820@gmail.com
>> > > >>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>> wrote:
>> > > >>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>> Hi folks,
>> > > >>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>> Just to provide some updates concerning the
>> > > >> status
>> > > >>>> on
>> > > >>>>>> the
>> > > >>>>>>>>>>>>>>>>> test instabilities.
>> > > >>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>> Currently, we have 30 unresolved tickets labeled
>> > > >>>> with
>> > > >>>>>>>>> `Affects
>> > > >>>>>>>>>>>>> Version`
>> > > >>>>>>>>>>>>>>>>> 1.11.x.
>> > > >>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>
>> > > >>>>>>>>>>>>
>> > > >>>>>>>>>>>
>> > > >>>>>>>>>>
>> > > >>>>>>>>>
>> > > >>>>>>>
>> > > >>>>>>
>> > > >>>>>
>> > > >>>>
>> > > >>>
>> > > >>
>> > >
>> >
>> https://issues.apache.org/jira/browse/FLINK-19775?filter=12348580&jql=project%20%3D%20FLINK%20AND%20resolution%20%3D%20Unresolved%20AND%20affectedVersion%20in%20(1.11.0%2C%201.11.1%2C%201.11.2%2C%201.11.3)%20AND%20labels%20%3D%20test-stability%20ORDER%20BY%20created%20DESC
>> > > >>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>> Among the 30 tickets, 11 of them are:
>> > > >>>>>>>>>>>>>>>>> - Have occured in the recent 3 months
>> > > >>>>>>>>>>>>>>>>> - Not confirmed to be pure testability issues
>> > > >>>>>>>>>>>>>>>>> - Not confirmed to be rare condition cases
>> > > >>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>> It would be nice if someone familiar with these
>> > > >>>>>> components
>> > > >>>>>>>>> can
>> > > >>>>>>>>>>>> take a
>> > > >>>>>>>>>>>>>>> look
>> > > >>>>>>>>>>>>>>>>> into these issues.
>> > > >>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>> -
>> > > >>> https://issues.apache.org/jira/browse/FLINK-17159
>> > > >>>>>> (ES6)
>> > > >>>>>>>>>>>>>>>>> -
>> > > >>> https://issues.apache.org/jira/browse/FLINK-17912
>> > > >>>>>>> (Kafka)
>> > > >>>>>>>>>>>>>>>>> -
>> > > >>> https://issues.apache.org/jira/browse/FLINK-17949
>> > > >>>>>>> (Kafka)
>> > > >>>>>>>>>>>>>>>>> ⁃
>> > > >>> https://issues.apache.org/jira/browse/FLINK-18444
>> > > >>>>>>> (Kafka)
>> > > >>>>>>>>>>>>>>>>> -
>> > > >>> https://issues.apache.org/jira/browse/FLINK-18634
>> > > >>>>>>> (Kafka)
>> > > >>>>>>>>>>>>>>>>> -
>> > > >>> https://issues.apache.org/jira/browse/FLINK-18648
>> > > >>>>>>> (Kafka)
>> > > >>>>>>>>>>>>>>>>> -
>> > > >>> https://issues.apache.org/jira/browse/FLINK-18807
>> > > >>>>>>> (Kafka)
>> > > >>>>>>>>>>>>>>>>> -
>> > > >>> https://issues.apache.org/jira/browse/FLINK-19369
>> > > >>>>>>>>>>>> (BlobClientTest)
>> > > >>>>>>>>>>>>>>>>> -
>> > > >>> https://issues.apache.org/jira/browse/FLINK-19436
>> > > >>>>>>> (TPCDS)
>> > > >>>>>>>>>>>>>>>>> -
>> > > >>> https://issues.apache.org/jira/browse/FLINK-19690
>> > > >>>>>>>>>>>> (Format/Parquet)
>> > > >>>>>>>>>>>>>>>>> -
>> > > >>> https://issues.apache.org/jira/browse/FLINK-19775
>> > > >>>>>>>>>>>>>>>>> (SystemProcessingTimeServiceTest)
>> > > >>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>> Thank you~
>> > > >>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>> Xintong Song
>> > > >>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>> On Thu, Oct 29, 2020 at 10:21 AM Jingsong Li <
>> > > >>>>>>>>>>>> jingsonglee0@gmail.com
>> > > >>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>> wrote:
>> > > >>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>> +1 to backport the FLIP-27 adjustments to
>> > > >>> 1.11.x.
>> > > >>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>> If possible, that would be great. Many people
>> > > >>> are
>> > > >>>>>>> looking
>> > > >>>>>>>>>>> forward
>> > > >>>>>>>>>>>>> to
>> > > >>>>>>>>>>>>>>> the
>> > > >>>>>>>>>>>>>>>>>> FLIP-27 interface, but they don't want to take
>> > > >>> the
>> > > >>>>>> risk
>> > > >>>>>>> to
>> > > >>>>>>>>>>>> upgrade
>> > > >>>>>>>>>>>>> to
>> > > >>>>>>>>>>>>>>>>> 1.12
>> > > >>>>>>>>>>>>>>>>>> (And wait 1.12). After all, 1.11 is a
>> > > >> relatively
>> > > >>>>>> stable
>> > > >>>>>>>>>>> version.
>> > > >>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>> Best,
>> > > >>>>>>>>>>>>>>>>>> Jingsong
>> > > >>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>> On Thu, Oct 29, 2020 at 1:24 AM Stephan Ewen <
>> > > >>>>>>>>>> sewen@apache.org
>> > > >>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>> wrote:
>> > > >>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>> Thanks for starting this.
>> > > >>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>> +1 form my side to backport the FLIP-27
>> > > >>>>> adjustments
>> > > >>>>>> to
>> > > >>>>>>>>>>> 1.11.x.
>> > > >>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>> There were quite a few changes, and I think
>> > > >> we
>> > > >>>>> need
>> > > >>>>>> to
>> > > >>>>>>>>>>>>> cherry-pick
>> > > >>>>>>>>>>>>>>>>> them
>> > > >>>>>>>>>>>>>>>>>> all
>> > > >>>>>>>>>>>>>>>>>>> to not get some inconsistent mix of changes
>> > > >>> and
>> > > >>>>> many
>> > > >>>>>>>>> merge
>> > > >>>>>>>>>>>>>>> conflicts.
>> > > >>>>>>>>>>>>>>>>>>> I made a list below of what we need to add
>> > > >> to
>> > > >>>>>>>>>> "release-1.11".
>> > > >>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>> *#### Core Source API Changes to backport
>> > > >> (in
>> > > >>>>>> REVERSE
>> > > >>>>>>>>>> order)*
>> > > >>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>  (Use: "git log
>> > > >>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>
>> > > >>>>> flink-core/src/main/java/org/apache/flink/api/connector/source")
>> > > >>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>> commit
>> > > >>> 162c072e9265a7b6dd9d6f5459eb7974091c4c4e
>> > > >>>>>>>>>>>>>>>>>>> [FLINK-19492][core] Consolidate Source
>> > > >> Events
>> > > >>>>>> between
>> > > >>>>>>>>>> Source
>> > > >>>>>>>>>>>> API
>> > > >>>>>>>>>>>>>>> and
>> > > >>>>>>>>>>>>>>>>>> Split
>> > > >>>>>>>>>>>>>>>>>>> Reader API
>> > > >>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>> commit
>> > > >>> 428fbbad85379f71a9fbf1a1ab3e3dcfa9006436
>> > > >>>>>>>>>>>>>>>>>>> [FLINK-19205][core] Add access to
>> > > >>> configuration
>> > > >>>>> and
>> > > >>>>>>>>>> hostname
>> > > >>>>>>>>>>> in
>> > > >>>>>>>>>>>>> the
>> > > >>>>>>>>>>>>>>>>>>> SourceReaderContext
>> > > >>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>> commit
>> > > >>> 723e1790744ddc78d19c7c978442af1383f38d33
>> > > >>>>>>>>>>>>>>>>>>> [hotfix][core] Add to Source Enumerator
>> > > >>>>> convenience
>> > > >>>>>>>>> methods
>> > > >>>>>>>>>>> to
>> > > >>>>>>>>>>>>>>> assign
>> > > >>>>>>>>>>>>>>>>>>> single split
>> > > >>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>> *#### Connector Base Changes to Backport (in
>> > > >>>>> REVERSE
>> > > >>>>>>>>>> order)*
>> > > >>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>  (Use: "git log
>> > > >>>>>>> flink-connectors/flink-connector-base")
>> > > >>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>> commit
>> > > >>> 401f56fe9d6b0271260edf9787cdcbfe4d03874d
>> > > >>>>>>>>>>>>>>>>>>> [FLINK-19427][FLINK-19489][tests] Fix test
>> > > >>>>>> conditions
>> > > >>>>>>>>> for
>> > > >>>>>>>>>>>>>>>>>>>
>> > > >>>>>>> 'SplitFetcherTest.testNotifiesWhenGoingIdleConcurrent()'
>> > > >>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>> commit
>> > > >>> 68c5c2ff779d82a1ff81ffaf60d8a1b283797db1
>> > > >>>>>>>>>>>>>>>>>>> [FLINK-19448][connector base] Explicitly
>> > > >> check
>> > > >>>> for
>> > > >>>>>>>>>>> un-expected
>> > > >>>>>>>>>>>>>>>>> condition
>> > > >>>>>>>>>>>>>>>>>>> that would leave an inconsistent state
>> > > >>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>> commit
>> > > >>> 162c072e9265a7b6dd9d6f5459eb7974091c4c4e
>> > > >>>>>>>>>>>>>>>>>>> [FLINK-19492][core] Consolidate Source
>> > > >> Events
>> > > >>>>>> between
>> > > >>>>>>>>>> Source
>> > > >>>>>>>>>>>> API
>> > > >>>>>>>>>>>>>>> and
>> > > >>>>>>>>>>>>>>>>>> Split
>> > > >>>>>>>>>>>>>>>>>>> Reader API
>> > > >>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>> commit
>> > > >>> c1ca7a4c7c21ec8868c14cdddd43c559625b794c
>> > > >>>>>>>>>>>>>>>>>>> [refactor][tests] Move some source test
>> > > >> utils
>> > > >>>> from
>> > > >>>>>>>>>>>>>>>>> flink-connector-base
>> > > >>>>>>>>>>>>>>>>>> to
>> > > >>>>>>>>>>>>>>>>>>> flink-core
>> > > >>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>> commit
>> > > >>> ee5c4c211c35c70d28252363bbc8400453609977
>> > > >>>>>>>>>>>>>>>>>>> [FLINK-19251][connectors] Avoid confusing
>> > > >>> queue
>> > > >>>>>>>>> handling in
>> > > >>>>>>>>>>>>>>>>>>> "SplitReader.handleSplitsChanges()"
>> > > >>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>> commit
>> > > >>> 5abef56b2bf85bcac786f6b16b6899b6cced7176
>> > > >>>>>>>>>>>>>>>>>>> [FLINK-19250][connectors] Fix error
>> > > >>> propagation
>> > > >>>> in
>> > > >>>>>>>>>> connector
>> > > >>>>>>>>>>>> base
>> > > >>>>>>>>>>>>>>>>>>> (SplitFetcherManager).
>> > > >>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>> commit
>> > > >>> 8fcca837c55a9216595ee4c03038b52747098dbb
>> > > >>>>>>>>>>>>>>>>>>> [hotfix][connectors] Improve JavaDocs for
>> > > >>>>>>>>>>>>>>> SingleThreadFetcherManager
>> > > >>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>> commit
>> > > >>> 4700bb5dde3303cbe98882f6beb7379425717b01
>> > > >>>>>>>>>>>>>>>>>>> [FLINK-19225][connectors] Various small
>> > > >>>>> improvements
>> > > >>>>>>> to
>> > > >>>>>>>>>>>>>>>>> SourceReaderBase
>> > > >>>>>>>>>>>>>>>>>>> (part 2)
>> > > >>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>> commit
>> > > >>> 12261c6b7ed6478a9b9f6a69cb58246b83cab9b7
>> > > >>>>>>>>>>>>>>>>>>> [FLINK-17393][connectors] (follow-up) Wakeup
>> > > >>> the
>> > > >>>>>>>>>>> SplitFetchers
>> > > >>>>>>>>>>>>> more
>> > > >>>>>>>>>>>>>>>>>>> elegantly.
>> > > >>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>> commit
>> > > >>> c60aaff0249bfd6b5871b7f82e03efc487a54d6b
>> > > >>>>>>>>>>>>>>>>>>> [hotfix][tests] Extend test coverage for
>> > > >>>>>>>>>>>>>>> FutureCompletingBlockingQueue
>> > > >>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>> commit
>> > > >>> cef8a587d7fd2fe64cc644da5ed095d82e46f631
>> > > >>>>>>>>>>>>>>>>>>> [FLINK-19245][connectors] Set default
>> > > >> capacity
>> > > >>>> for
>> > > >>>>>>>>>>>>>>>>>>> FutureCompletingBlockingQueue.
>> > > >>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>> commit
>> > > >>> 4ea95782b4c6a2538153d4d16ad3f4839c7de0fb
>> > > >>>>>>>>>>>>>>>>>>> [FLINK-19223][connectors] Simplify
>> > > >>> Availability
>> > > >>>>>> Future
>> > > >>>>>>>>>> Model
>> > > >>>>>>>>>>> in
>> > > >>>>>>>>>>>>>>> Base
>> > > >>>>>>>>>>>>>>>>>>> Connector
>> > > >>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>> commit
>> > > >>> 511857049ba30c8ff0ee56da551fa4a479dc583e
>> > > >>>>>>>>>>>>>>>>>>> [FLINK-18128][connectors] Ensure idle split
>> > > >>>>> fetchers
>> > > >>>>>>>>> lead
>> > > >>>>>>>>>> to
>> > > >>>>>>>>>>>>>>>>> availability
>> > > >>>>>>>>>>>>>>>>>>> notifications.
>> > > >>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>> commit
>> > > >>> a8206467af0830dcb89623ea068b5ca3b3450c92
>> > > >>>>>>>>>>>>>>>>>>> [refactor][core] Eagerly initialize the
>> > > >>>> FetchTask
>> > > >>>>> to
>> > > >>>>>>>>>> support
>> > > >>>>>>>>>>>>> proper
>> > > >>>>>>>>>>>>>>>>> unit
>> > > >>>>>>>>>>>>>>>>>>> testing
>> > > >>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>> commit
>> > > >>> 3b2f54bcb437f98e6137c904045cc51072b5c06b
>> > > >>>>>>>>>>>>>>>>>>> [hotfix][tests] Move constants in
>> > > >>>> SplitFetcherTest
>> > > >>>>>>>>> relevant
>> > > >>>>>>>>>>> to
>> > > >>>>>>>>>>>>> only
>> > > >>>>>>>>>>>>>>>>> one
>> > > >>>>>>>>>>>>>>>>>>> test into test method
>> > > >>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>> commit
>> > > >>> d7625760a75a508bf05bcddc380bb4d62ee1743e
>> > > >>>>>>>>>>>>>>>>>>> [FLINK-19225][connectors] Various small
>> > > >>>>> improvements
>> > > >>>>>>> to
>> > > >>>>>>>>>>>>>>>>> SourceReaderBase
>> > > >>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>> commit
>> > > >>> a5b0d3297748c1be47ad579a88f24df2255a8df1
>> > > >>>>>>>>>>>>>>>>>>> [FLINK-17393][connectors] Wakeup the
>> > > >>>> SplitFetchers
>> > > >>>>>>> more
>> > > >>>>>>>>>>>>> elegantly.
>> > > >>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>> commit
>> > > >>> f42a3ebc3e81a034b7221a803c153636fef34903
>> > > >>>>>>>>>>>>>>>>>>> [FLINK-18680][connectors] Make connector
>> > > >> base
>> > > >>>>>>>>>>>> RecordsWithSplitIds
>> > > >>>>>>>>>>>>>>> more
>> > > >>>>>>>>>>>>>>>>>>> lightweight.
>> > > >>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>> commit
>> > > >>> e3d273de822b085183d09b275a445879ff94b350
>> > > >>>>>>>>>>>>>>>>>>> [FLINK-19162][connectors] Add 'recycle()' to
>> > > >>> the
>> > > >>>>>>>>>>>>>>> RecordsWithSplitIds
>> > > >>>>>>>>>>>>>>>>> to
>> > > >>>>>>>>>>>>>>>>>>> support reuse of heavy objects.
>> > > >>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>> commit
>> > > >>> 8ebc464c2520453a70001cd712abc8dee6ee89e0
>> > > >>>>>>>>>>>>>>>>>>> [hotfix][testing] Add a set of
>> > > >> parameterizable
>> > > >>>>>> testing
>> > > >>>>>>>>>> mocks
>> > > >>>>>>>>>>>> for
>> > > >>>>>>>>>>>>>>> the
>> > > >>>>>>>>>>>>>>>>>> Split
>> > > >>>>>>>>>>>>>>>>>>> Reader API
>> > > >>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>> commit
>> > > >>> 930a07438be1185388d7150640f294dfe2a5d378
>> > > >>>>>>>>>>>>>>>>>>> [hotfix][connectors] Add RequestSplitEvent
>> > > >> to
>> > > >>>>>>>>>>>>>>> 'flink-connector-base'
>> > > >>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>> commit
>> > > >>> e945ce8a933bc378844782f784ca473c767ca159
>> > > >>>>>>>>>>>>>>>>>>> [refactor][connectors] Backport of the
>> > > >>>>>> connector-base
>> > > >>>>>>>>>>> exception
>> > > >>>>>>>>>>>>>>>>> handling
>> > > >>>>>>>>>>>>>>>>>>> from the Kafka Connector Pull Request
>> > > >>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>> On Wed, Oct 28, 2020 at 5:49 PM Steven Wu <
>> > > >>>>>>>>>>>> stevenz3wu@gmail.com>
>> > > >>>>>>>>>>>>>>>>> wrote:
>> > > >>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>> I would love to see this FLIP-27 source
>> > > >>>>> interface
>> > > >>>>>>>>>>> improvement
>> > > >>>>>>>>>>>>> [1]
>> > > >>>>>>>>>>>>>>>>> made
>> > > >>>>>>>>>>>>>>>>>> to
>> > > >>>>>>>>>>>>>>>>>>>> 1.11.3.
>> > > >>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>> [1]
>> > > >>>>>>> https://issues.apache.org/jira/browse/FLINK-19698
>> > > >>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>> On Wed, Oct 28, 2020 at 12:32 AM Tzu-Li
>> > > >>>> (Gordon)
>> > > >>>>>>> Tai <
>> > > >>>>>>>>>>>>>>>>>>> tzulitai@apache.org>
>> > > >>>>>>>>>>>>>>>>>>>> wrote:
>> > > >>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>> Thanks for the replies so far!
>> > > >>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>> Just to provide a brief update on the
>> > > >>> status
>> > > >>>>> of
>> > > >>>>>>>>>> blockers
>> > > >>>>>>>>>>>> for
>> > > >>>>>>>>>>>>>>>>> 1.11.3
>> > > >>>>>>>>>>>>>>>>>> so
>> > > >>>>>>>>>>>>>>>>>>>> far:
>> > > >>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>> *PR opened, pending reviewer*-
>> > > >>> [FLINK-19717]
>> > > >>>>>>>>>>>>>>>>>> SourceReaderBase.pollNext
>> > > >>>>>>>>>>>>>>>>>>>> may
>> > > >>>>>>>>>>>>>>>>>>>>> return END_OF_INPUT if SplitReader.fetch
>> > > >>>>> throws
>> > > >>>>>> (
>> > > >>>>>>>>>>>>>>>>>>>>>
>> > > >>> https://github.com/apache/flink/pull/13776)
>> > > >>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>> *PR opened, reviewed + close to being
>> > > >>>> merged*
>> > > >>>>>>>>>>>>>>>>>>>>> - [FLINK-19741] Timer service should
>> > > >> skip
>> > > >>>>>>> restoring
>> > > >>>>>>>>>> from
>> > > >>>>>>>>>>>> raw
>> > > >>>>>>>>>>>>>>> keyed
>> > > >>>>>>>>>>>>>>>>>>> stream
>> > > >>>>>>>>>>>>>>>>>>>>> if it isn't the writer (
>> > > >>>>>>>>>>>>>>>>> https://github.com/apache/flink/pull/13761)
>> > > >>>>>>>>>>>>>>>>>>>>> - [FLINK-19748] Raw keyed stream key
>> > > >> group
>> > > >>>>>>> iterator
>> > > >>>>>>>>>>> should
>> > > >>>>>>>>>>>> be
>> > > >>>>>>>>>>>>>>>>>> skipping
>> > > >>>>>>>>>>>>>>>>>>>>> unwritten key groups (
>> > > >>>>>>>>>>>>>>> https://github.com/apache/flink/pull/13772)
>> > > >>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>> *Merged*
>> > > >>>>>>>>>>>>>>>>>>>>> - [FLINK-19154] Application mode deletes
>> > > >>> HA
>> > > >>>>> data
>> > > >>>>>>> in
>> > > >>>>>>>>>> case
>> > > >>>>>>>>>>> of
>> > > >>>>>>>>>>>>>>>>> suspended
>> > > >>>>>>>>>>>>>>>>>>>>> ZooKeeper connection
>> > > >>>>>>>>>>>>>>>>>>>>> - [FLINK-19569] Upgrade ICU4J to 67.1+
>> > > >>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>> Right now as it seems, progress is
>> > > >> mainly
>> > > >>>>>> blocked
>> > > >>>>>>>>> on a
>> > > >>>>>>>>>>>>> reviewer
>> > > >>>>>>>>>>>>>>>>> for
>> > > >>>>>>>>>>>>>>>>>>>>> FLINK-19717.
>> > > >>>>>>>>>>>>>>>>>>>>> Meanwhile, Xintong is keeping an eye on
>> > > >>> test
>> > > >>>>>>>>>>> instabilities
>> > > >>>>>>>>>>>>> [1]
>> > > >>>>>>>>>>>>>>> to
>> > > >>>>>>>>>>>>>>>>> see
>> > > >>>>>>>>>>>>>>>>>>> if
>> > > >>>>>>>>>>>>>>>>>>>>> there are any fixes that should be
>> > > >> applied
>> > > >>>> to
>> > > >>>>>>>>>>>> `release-1.11`.
>> > > >>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>> This is also a reminder, that if there
>> > > >> are
>> > > >>>>> other
>> > > >>>>>>>>>> blockers
>> > > >>>>>>>>>>>>> that
>> > > >>>>>>>>>>>>>>> we
>> > > >>>>>>>>>>>>>>>>>> need
>> > > >>>>>>>>>>>>>>>>>>> to
>> > > >>>>>>>>>>>>>>>>>>>>> be aware of, or a need to re-establish
>> > > >>>>> estimated
>> > > >>>>>>>>> time
>> > > >>>>>>>>>> for
>> > > >>>>>>>>>>>>>>> getting
>> > > >>>>>>>>>>>>>>>>>> fixes
>> > > >>>>>>>>>>>>>>>>>>>> in
>> > > >>>>>>>>>>>>>>>>>>>>> and delay the RC for 1.11.3, please do
>> > > >> let
>> > > >>>> us
>> > > >>>>>>> know!
>> > > >>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>> Cheers,
>> > > >>>>>>>>>>>>>>>>>>>>> Gordon
>> > > >>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>> [1]
>> > > >>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>
>> > > >>>>>>>>>>>>
>> > > >>>>>>>>>>>
>> > > >>>>>>>>>>
>> > > >>>>>>>>>
>> > > >>>>>>>
>> > > >>>>>>
>> > > >>>>>
>> > > >>>>
>> > > >>>
>> > > >>
>> > >
>> >
>> https://issues.apache.org/jira/issues/?jql=project%20%3D%20FLINK%20AND%20status%20in%20(Open%2C%20%22In%20Progress%22%2C%20Reopened)%20AND%20affectedVersion%20in%20(1.11.0%2C%201.11.1%2C%201.11.2)%20AND%20labels%20%3D%20test-stability
>> > > >>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>> On Mon, Oct 26, 2020 at 9:43 PM Kostas
>> > > >>>>> Kloudas <
>> > > >>>>>>>>>>>>>>>>> kkloudas@apache.org>
>> > > >>>>>>>>>>>>>>>>>>>>> wrote:
>> > > >>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>> +1 for releasing Flink 1.11.3 as it
>> > > >>>>> contains a
>> > > >>>>>>>>> number
>> > > >>>>>>>>>>> of
>> > > >>>>>>>>>>>>>>>>> important
>> > > >>>>>>>>>>>>>>>>>>>>>> fixes and thanks Gordon and Xintong
>> > > >> for
>> > > >>>>>>>>> volunteering.
>> > > >>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>> Cheers,
>> > > >>>>>>>>>>>>>>>>>>>>>> Kostas
>> > > >>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>> On Mon, Oct 26, 2020 at 4:37 AM Yu Li
>> > > >> <
>> > > >>>>>>>>>>> carp84@gmail.com>
>> > > >>>>>>>>>>>>>>> wrote:
>> > > >>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>> +1 for releasing Flink 1.11.3, and
>> > > >>>> thanks
>> > > >>>>>>> Gordon
>> > > >>>>>>>>>> and
>> > > >>>>>>>>>>>>>>> Xintong
>> > > >>>>>>>>>>>>>>>>> for
>> > > >>>>>>>>>>>>>>>>>>>>>>> volunteering as our release
>> > > >> managers.
>> > > >>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>> Best Regards,
>> > > >>>>>>>>>>>>>>>>>>>>>>> Yu
>> > > >>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>> On Mon, 26 Oct 2020 at 09:45,
>> > > >> Xintong
>> > > >>>>> Song <
>> > > >>>>>>>>>>>>>>>>>> tonysong820@gmail.com>
>> > > >>>>>>>>>>>>>>>>>>>>>> wrote:
>> > > >>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>> Thanks Gordan for starting this
>> > > >>>>>> discussion.
>> > > >>>>>>>>>>>>>>>>>>>>>>>> My pleasure to help with the
>> > > >> release
>> > > >>>>>>> process.
>> > > >>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>> Thank you~
>> > > >>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>> Xintong Song
>> > > >>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>> On Fri, Oct 23, 2020 at 11:29 PM
>> > > >>> Till
>> > > >>>>>>>>> Rohrmann <
>> > > >>>>>>>>>>>>>>>>>>>> trohrmann@apache.org
>> > > >>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>> wrote:
>> > > >>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>> Thanks for starting this
>> > > >>> discussion
>> > > >>>>>>> Gordon.
>> > > >>>>>>>>>> There
>> > > >>>>>>>>>>>> are
>> > > >>>>>>>>>>>>>>> over
>> > > >>>>>>>>>>>>>>>>>> 100
>> > > >>>>>>>>>>>>>>>>>>>>> issues
>> > > >>>>>>>>>>>>>>>>>>>>>>>>> which are fixed for 1.11.3.
>> > > >> Hence
>> > > >>> +1
>> > > >>>>>> for a
>> > > >>>>>>>>>>> soonish
>> > > >>>>>>>>>>>>>>> 1.11.3
>> > > >>>>>>>>>>>>>>>>>>>> release.
>> > > >>>>>>>>>>>>>>>>>>>>>> Thanks
>> > > >>>>>>>>>>>>>>>>>>>>>>>>> for volunteering as our release
>> > > >>>>> managers
>> > > >>>>>>>>> Gordon
>> > > >>>>>>>>>>> and
>> > > >>>>>>>>>>>>>>>>> Xintong!
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>> Cheers,
>> > > >>>>>>>>>>>>>>>>>>>>>>>>> Till
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>> On Fri, Oct 23, 2020 at 5:02 PM
>> > > >>>> Tzu-Li
>> > > >>>>>>>>> (Gordon)
>> > > >>>>>>>>>>>> Tai <
>> > > >>>>>>>>>>>>>>>>>>>>>> tzulitai@apache.org
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>> Hi,
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>> Xintong and I would like to
>> > > >>> start a
>> > > >>>>>>>>> discussion
>> > > >>>>>>>>>>> for
>> > > >>>>>>>>>>>>>>>>> releasing
>> > > >>>>>>>>>>>>>>>>>>>> Flink
>> > > >>>>>>>>>>>>>>>>>>>>>>>> 1.11.3
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>> soon.
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>> It seems like we already have a
>> > > >>> few
>> > > >>>>>>>>> pressing
>> > > >>>>>>>>>>>> issues
>> > > >>>>>>>>>>>>>>> that
>> > > >>>>>>>>>>>>>>>>>> needs
>> > > >>>>>>>>>>>>>>>>>>>> to
>> > > >>>>>>>>>>>>>>>>>>>>> be
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>> included in a new hotfix
>> > > >> release:
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>   - Heap-based timers’ restore
>> > > >>>>>> behaviour
>> > > >>>>>>>>> is
>> > > >>>>>>>>>>>>> causing a
>> > > >>>>>>>>>>>>>>>>>>> critical
>> > > >>>>>>>>>>>>>>>>>>>>>> recovery
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>   issue for StateFun [1] [2]
>> > > >>> [3].
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>   - There are several
>> > > >> robustness
>> > > >>>>>> issues
>> > > >>>>>>>>> for
>> > > >>>>>>>>>> the
>> > > >>>>>>>>>>>>>>> FLIP-27
>> > > >>>>>>>>>>>>>>>>> new
>> > > >>>>>>>>>>>>>>>>>>>>> source
>> > > >>>>>>>>>>>>>>>>>>>>>> API,
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>   such as [4]. We already have
>> > > >>>> some
>> > > >>>>>>> users
>> > > >>>>>>>>>> using
>> > > >>>>>>>>>>>> the
>> > > >>>>>>>>>>>>>>>>> FLIP-27
>> > > >>>>>>>>>>>>>>>>>>> API
>> > > >>>>>>>>>>>>>>>>>>>>>> with
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>> 1.11.x,
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>   so it would be important to
>> > > >>> get
>> > > >>>>>> those
>> > > >>>>>>>>> fixes
>> > > >>>>>>>>>>> in
>> > > >>>>>>>>>>>>> for
>> > > >>>>>>>>>>>>>>>>> 1.11.x
>> > > >>>>>>>>>>>>>>>>>>> as
>> > > >>>>>>>>>>>>>>>>>>>>>> well.
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>> Apart from the issues that are
>> > > >>>>> already
>> > > >>>>>>>>> marked
>> > > >>>>>>>>>> as
>> > > >>>>>>>>>>>>>>> blocker
>> > > >>>>>>>>>>>>>>>>> for
>> > > >>>>>>>>>>>>>>>>>>>>> 1.11.3
>> > > >>>>>>>>>>>>>>>>>>>>>> in
>> > > >>>>>>>>>>>>>>>>>>>>>>>> our
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>> JIRA [5], please let us know in
>> > > >>>> this
>> > > >>>>>>>>> thread if
>> > > >>>>>>>>>>>> there
>> > > >>>>>>>>>>>>>>> is
>> > > >>>>>>>>>>>>>>>>>>> already
>> > > >>>>>>>>>>>>>>>>>>>>>> ongoing
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>> work for other important fixes
>> > > >>> that
>> > > >>>>> we
>> > > >>>>>>>>> should
>> > > >>>>>>>>>>> try
>> > > >>>>>>>>>>>> to
>> > > >>>>>>>>>>>>>>>>>> include.
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>> Xintong and I would like to
>> > > >>>> volunteer
>> > > >>>>>> for
>> > > >>>>>>>>>>> managing
>> > > >>>>>>>>>>>>>>> this
>> > > >>>>>>>>>>>>>>>>>>> release,
>> > > >>>>>>>>>>>>>>>>>>>>> and
>> > > >>>>>>>>>>>>>>>>>>>>>>>> will
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>> try to communicate the priority
>> > > >>> of
>> > > >>>>>>> pending
>> > > >>>>>>>>>>>> blockers
>> > > >>>>>>>>>>>>>>> over
>> > > >>>>>>>>>>>>>>>>> the
>> > > >>>>>>>>>>>>>>>>>>>> next
>> > > >>>>>>>>>>>>>>>>>>>>>> few
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>> days.
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>> Since the aforementioned issues
>> > > >>> are
>> > > >>>>>> quite
>> > > >>>>>>>>>>>> critical,
>> > > >>>>>>>>>>>>>>> we’d
>> > > >>>>>>>>>>>>>>>>>> like
>> > > >>>>>>>>>>>>>>>>>>> to
>> > > >>>>>>>>>>>>>>>>>>>>> aim
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>> for a *feature
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>> freeze by the end of next week
>> > > >>>> (Oct.
>> > > >>>>>>> 30th)*
>> > > >>>>>>>>>> and
>> > > >>>>>>>>>>>>> start
>> > > >>>>>>>>>>>>>>> the
>> > > >>>>>>>>>>>>>>>>>>>> release
>> > > >>>>>>>>>>>>>>>>>>>>>> voting
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>> process the week after.
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>> If that is too short of a
>> > > >> notice
>> > > >>>> and
>> > > >>>>>> you
>> > > >>>>>>>>> might
>> > > >>>>>>>>>>>> need
>> > > >>>>>>>>>>>>>>> more
>> > > >>>>>>>>>>>>>>>>>> time,
>> > > >>>>>>>>>>>>>>>>>>>>>> please
>> > > >>>>>>>>>>>>>>>>>>>>>>>> let
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>> us know!
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>> Cheers,
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>> Gordon
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>> [1]
>> > > >>>>>>>>>>>>> https://issues.apache.org/jira/browse/FLINK-19692
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>> [2]
>> > > >>>>>>>>>>>>> https://issues.apache.org/jira/browse/FLINK-19741
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>> [3]
>> > > >>>>>>>>>>>>> https://issues.apache.org/jira/browse/FLINK-19748
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>> [4]
>> > > >>>>>>>>>>>>> https://issues.apache.org/jira/browse/FLINK-19717
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>> [5]
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>
>> > > >>>>>>>>>>>>
>> > > >>>>>>>>>>>
>> > > >>>>>>>>>>
>> > > >>>>>>>>>
>> > > >>>>>>>
>> > > >>>>>>
>> > > >>>>>
>> > > >>>>
>> > > >>>
>> > > >>
>> > >
>> >
>> https://issues.apache.org/jira/issues/?jql=project%20%3D%20FLINK%20AND%20priority%20%3D%20Blocker%20AND%20fixVersion%20%3D%201.11.3
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>> --
>> > > >>>>>>>>>>>>>>>>>> Best, Jingsong Lee
>> > > >>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>
>> > > >>>>>>>>>>>>
>> > > >>>>>>>>>>>
>> > > >>>>>>>>>>
>> > > >>>>>>>>>
>> > > >>>>>>>>
>> > > >>>>>>>
>> > > >>>>>>
>> > > >>>>>
>> > > >>>>
>> > > >>>
>> > > >>
>> > >
>> > >
>> > >
>> >
>>
>

Re: [DISCUSS] Releasing Apache Flink 1.11.3

Posted by Xintong Song <to...@gmail.com>.
Thanks for the notice and fix, Roman.

Thank you~

Xintong Song



On Wed, Nov 11, 2020 at 5:53 PM Khachatryan Roman <
khachatryan.roman@gmail.com> wrote:

> Hi,
>
> I'd like FLINK-20079 [1] to be merged into 1.11 and included in 1.11.3.
>
> [1] https://issues.apache.org/jira/browse/FLINK-20079
>
> Regards,
> Roman
>
>
> On Tue, Nov 10, 2020 at 8:21 AM Xintong Song <to...@gmail.com>
> wrote:
>
> > Thanks for the notice, Dian.
> >
> > Thank you~
> >
> > Xintong Song
> >
> >
> >
> > On Tue, Nov 10, 2020 at 10:19 AM Dian Fu <di...@gmail.com> wrote:
> >
> > > Hi Xintong,
> > >
> > > I want to bring one more issue to your attention [1]. The test case
> > > UnalignedCheckpointCompatibilityITCase.test failed several times in the
> > > last nightly test of release-1.11. We need to figure out if it's just
> an
> > > instable test or caused by recent changes.
> > >
> > > [1] https://issues.apache.org/jira/browse/FLINK-20065
> > >
> > > > 在 2020年11月10日，上午9:24，Xintong Song <to...@gmail.com> 写道：
> > > >
> > > > Thanks for the replies.
> > > >
> > > > Thank you~
> > > >
> > > > Xintong Song
> > > >
> > > >
> > > >
> > > > On Tue, Nov 10, 2020 at 1:09 AM Becket Qin <be...@gmail.com>
> > wrote:
> > > >
> > > >> Hi Xintong,
> > > >>
> > > >> Thanks for driving the release. Just want to sync up on the FLIP-27
> > > >> backporting. Stephan and I are still trying to backport a bunch of
> > > patches
> > > >> of Source to 1.11.3. Including:
> > > >>
> > > >> [FLINK-19698][connector/common] Add a close() method to the
> > SplitReader.
> > > >> [FLINK-19717] SourceReaderBase.pollNext may return END_OF_INPUT if
> > > >> SplitReader.fetch throws
> > > >> [FLINK-19535] [connector/common] Avoid failing a job multiple times
> in
> > > >> SourceCoordinator.
> > > >> [FLINK-19265] [FLINK-20049][core] Source API final adjustments.
> > > >>
> > > >> and a few more fixes.
> > > >>
> > > >> We are currently trying to fix them in 1.12 first so it might take a
> > > little
> > > >> longer to backport them to 1.11.3. I think it will probably take us
> a
> > > few
> > > >> more days to finish the backport. So that would roughly be the end
> of
> > > this
> > > >> week.
> > > >>
> > > >> Thanks,
> > > >>
> > > >> Jiangjie (Becket) Qin
> > > >>
> > > >>
> > > >>
> > > >>
> > > >> On Mon, Nov 9, 2020 at 9:57 PM Till Rohrmann <tr...@apache.org>
> > > wrote:
> > > >>
> > > >>> Yes, I've downgraded FLINK-19816 to critical.
> > > >>>
> > > >>> Cheers,
> > > >>> Till
> > > >>>
> > > >>> On Mon, Nov 9, 2020 at 10:19 AM Xintong Song <
> tonysong820@gmail.com>
> > > >>> wrote:
> > > >>>
> > > >>>> Thanks for the notice, Till.
> > > >>>>
> > > >>>> I just checked and found FLINK-20033 is already fixed. Shall we
> also
> > > >>>> downgrade FLINK-19816 to `Critical`?
> > > >>>>
> > > >>>> Thank you~
> > > >>>>
> > > >>>> Xintong Song
> > > >>>>
> > > >>>>
> > > >>>>
> > > >>>> On Mon, Nov 9, 2020 at 4:42 PM Till Rohrmann <
> trohrmann@apache.org>
> > > >>> wrote:
> > > >>>>
> > > >>>>> I would like to bring one more critical issue to your attention
> > which
> > > >>> is
> > > >>>>> FLINK-20033 [1]. I believe that this issue is actually causing
> what
> > > >> has
> > > >>>>> been reported in FLINK-19816 [2]. I hope to have it fixed by the
> > end
> > > >> of
> > > >>>>> today. Once FLINK-20033 is fixed, I think that we don't have to
> > block
> > > >>> the
> > > >>>>> release on FLINK-19816.
> > > >>>>>
> > > >>>>> [1] https://issues.apache.org/jira/browse/FLINK-20033
> > > >>>>> [2] https://issues.apache.org/jira/browse/FLINK-19816
> > > >>>>>
> > > >>>>> Cheers,
> > > >>>>> Till
> > > >>>>>
> > > >>>>> On Mon, Nov 9, 2020 at 4:05 AM Xintong Song <
> tonysong820@gmail.com
> > >
> > > >>>> wrote:
> > > >>>>>
> > > >>>>>> Hi devs,
> > > >>>>>>
> > > >>>>>> I'd like to provide an update on the progress of preparing
> release
> > > >>>>> 1.11.3.
> > > >>>>>>
> > > >>>>>> *Blockers*
> > > >>>>>> We currently have 3 remaining blockers. (3 resolved and 1
> emerged
> > > >>>>> compared
> > > >>>>>> to last week)
> > > >>>>>>
> > > >>>>>> - [FLINK-19698] Add close() method and onCheckpointComplete() to
> > > >> the
> > > >>>>>> Source.
> > > >>>>>> The issue has been fixed on the master branch. It's currently
> > > >> blocked
> > > >>>> on
> > > >>>>>> the FLIP-27 backportings to backport it to the 1.11 branch.
> > > >>>>>>
> > > >>>>>> - [FLINK-19717] SourceReaderBase.pollNext may return
> END_OF_INPUT
> > > >> if
> > > >>>>>> SplitReader.fetch throws
> > > >>>>>> A PR has been opened and reviewed. From the discussions on the
> PR,
> > > >> it
> > > >>>>> looks
> > > >>>>>> close to mergeable.
> > > >>>>>>
> > > >>>>>> - [FLINK-19816] Flink restored from a wrong checkpoint (a very
> old
> > > >>> one
> > > >>>>> and
> > > >>>>>> not the last completed one)
> > > >>>>>> This is a newly emerged blocker and Matthias is working on it.
> > > >>>>>>
> > > >>>>>> *Test Instabilities*
> > > >>>>>> We currently have 27 test instabilities[1].
> > > >>>>>> AFAIK, none of them are as serious as to block the 1.11.3
> release.
> > > >>>>>>
> > > >>>>>> *FLIP-27 Backprotings*
> > > >>>>>>
> > > >>>>>> I noticed that there's no jira issues opened on the FLIP-27
> > > >>> backporting
> > > >>>>>> efforts, which is part of the major efforts planned for the
> 1.11.3
> > > >>>>> release,
> > > >>>>>> making it hard to track the progress.
> > > >>>>>>
> > > >>>>>>
> > > >>>>>> @Stephan and @Becket, could you please share the updates on the
> > > >>>>> backporting
> > > >>>>>> efforts? How is the progress and when are the efforts expected
> to
> > > >> be
> > > >>>>>> finished? It would be appreciated and helpful if we can have a
> > jira
> > > >>>>> ticket
> > > >>>>>> to track the backporing progress publicly.
> > > >>>>>>
> > > >>>>>>
> > > >>>>>> Thank you~
> > > >>>>>>
> > > >>>>>> Xintong Song
> > > >>>>>>
> > > >>>>>>
> > > >>>>>> [1]
> > > >>>>>>
> > > >>>>>>
> > > >>>>>
> > > >>>>
> > > >>>
> > > >>
> > >
> >
> https://issues.apache.org/jira/browse/FLINK-16908?filter=12348580&jql=project%20%3D%20FLINK%20AND%20resolution%20%3D%20Unresolved%20AND%20affectedVersion%20in%20(1.11.0%2C%201.11.1%2C%201.11.2%2C%201.11.3)%20AND%20labels%20%3D%20test-stability%20ORDER%20BY%20updated%20DESC%2C%20created%20DESC
> > > >>>>>>
> > > >>>>>> On Wed, Nov 4, 2020 at 3:58 PM Tzu-Li (Gordon) Tai <
> > > >>>> tzulitai@apache.org>
> > > >>>>>> wrote:
> > > >>>>>>
> > > >>>>>>> Thanks for the efforts so far with pushing for 1.11.3.
> > > >>>>>>>
> > > >>>>>>> I'd like to provide a mid-week update on how we're looking with
> > > >> all
> > > >>>> the
> > > >>>>>>> pending blockers and backports:
> > > >>>>>>>
> > > >>>>>>> *Blockers**:*
> > > >>>>>>>
> > > >>>>>>> Currently all blockers either have an open PR, or have been
> > > >> merged.
> > > >>>>> I'll
> > > >>>>>>> highlight below the blockers *that still require reviewing
> > > >> efforts
> > > >>>> for
> > > >>>>>> them
> > > >>>>>>> to move forward:*
> > > >>>>>>>
> > > >>>>>>> - [FLINK-19909] Flink application in attach mode could not
> > > >>> terminate
> > > >>>>> when
> > > >>>>>>> the only job is canceled. PR:
> > > >>>>> https://github.com/apache/flink/pull/13911
> > > >>>>>>> - [FLINK-19717] SourceReaderBase.pollNext may return
> END_OF_INPUT
> > > >>> if
> > > >>>>>>> SplitReader.fetch throws. PR:
> > > >>>>> https://github.com/apache/flink/pull/13776
> > > >>>>>>>
> > > >>>>>>> The above PRs currently have no reviews at all yet. They seem
> to
> > > >>>>> already
> > > >>>>>>> have designated reviewers.
> > > >>>>>>>
> > > >>>>>>>
> > > >>>>>>> *Backports:*
> > > >>>>>>> - FLIP-27 robustness improvement backports: Stephan is
> currently
> > > >>>>> working
> > > >>>>>> on
> > > >>>>>>> backporting several FLIP-27 changes. There are no PRs yet for
> the
> > > >>>>>>> backported changes.
> > > >>>>>>> - DataStreamUtils.collect() refactorings backport: Steven Wu
> > > >>>> mentioned
> > > >>>>> to
> > > >>>>>>> backport this, but AFAIK this isn't assigned to anyone yet.
> > > >>>>>>>
> > > >>>>>>> *ETAs*:
> > > >>>>>>>
> > > >>>>>>> I'd like to request ETAs for the remaining backports, to
> prevent
> > > >> a
> > > >>>>> creep
> > > >>>>>> in
> > > >>>>>>> the scope of this bugfix release.
> > > >>>>>>>
> > > >>>>>>> We already have Flink users that would benefit from fixes that
> > > >> have
> > > >>>>> been
> > > >>>>>>> merged to release-1.11 already, so technically speaking the
> > > >>> backports
> > > >>>>>>> should be considered "nice-to-have" (to the best of my
> knowledge
> > > >> of
> > > >>>> the
> > > >>>>>>> changes) and could potentially be moved to a follow-up 1.11.4.
> > > >>>>>>> Most notably, the Stateful Functions project is already waiting
> > > >> on
> > > >>>>> Flink
> > > >>>>>>> 1.11.3 to address critical recovery issues (please see the
> > > >> StateFun
> > > >>>>> 2.2.1
> > > >>>>>>> release discussion thread [1]).
> > > >>>>>>>
> > > >>>>>>> @Stephan Ewen <se...@apache.org> @Becket Qin <
> > > >> becket.qin@gmail.com
> > > >>>>
> > > >>>>>> could
> > > >>>>>>> you provide an ETA for the FLIP-27 backports? It would help to
> > > >> get
> > > >>> a
> > > >>>>>> better
> > > >>>>>>> estimate to decide how we proceed here.
> > > >>>>>>>
> > > >>>>>>> Cheers,
> > > >>>>>>> Gordon
> > > >>>>>>>
> > > >>>>>>> [1]
> > > >>>>>>>
> > > >>>>>>>
> > > >>>>>>
> > > >>>>>
> > > >>>>
> > > >>>
> > > >>
> > >
> >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Releasing-StateFun-hotfix-version-2-2-1-td46239.html
> > > >>>>>>>
> > > >>>>>>> On Wed, Nov 4, 2020 at 3:16 PM Tzu-Li (Gordon) Tai <
> > > >>>>> tzulitai@apache.org>
> > > >>>>>>> wrote:
> > > >>>>>>>
> > > >>>>>>>>> The collect() utils can be picked back, I see no issue with
> > > >>> that
> > > >>>>> (it
> > > >>>>>> is
> > > >>>>>>>> isolated utilities).
> > > >>>>>>>>
> > > >>>>>>>> Just checking on all the requested backports mentioned in this
> > > >>>>> thread,
> > > >>>>>>> and
> > > >>>>>>>> figuring out which ones seem to still be unassigned / open.
> > > >>>>>>>>
> > > >>>>>>>> Is someone working on backporting
> > > >>>>>>>>
> > > >>>>>>>
> > > >>>>>>
> > > >>>>>
> > > >>>>
> > > >>>
> > > >>
> > >
> >
> https://github.com/apache/flink/commit/09a7a66b7313fea64817fe960a8da1265b428efc
> > > >>>>>>>> to release-1.11 at the moment?
> > > >>>>>>>>
> > > >>>>>>>> On Wed, Nov 4, 2020 at 1:39 AM Steven Wu <
> stevenz3wu@gmail.com
> > > >>>
> > > >>>>> wrote:
> > > >>>>>>>>
> > > >>>>>>>>> @Stephan Ewen <se...@apache.org> yeah, we can do that. don't
> > > >>>> worry
> > > >>>>>>> about
> > > >>>>>>>>> it. your earlier email had the perfect explanation on why
> file
> > > >>>>> source
> > > >>>>>>>>> shouldn't be backported.
> > > >>>>>>>>>
> > > >>>>>>>>> On Tue, Nov 3, 2020 at 3:37 AM Stephan Ewen <
> sewen@apache.org
> > > >>>
> > > >>>>> wrote:
> > > >>>>>>>>>
> > > >>>>>>>>>> @Steven would it be possible to initially copy some of the
> > > >>> code
> > > >>>>> into
> > > >>>>>>> the
> > > >>>>>>>>>> iceberg source and later replace it by a dependency on the
> > > >>> Flink
> > > >>>>>> file
> > > >>>>>>>>>> source?
> > > >>>>>>>>>>
> > > >>>>>>>>>> On Mon, Nov 2, 2020 at 8:33 PM Steven Wu <
> > > >>> stevenz3wu@gmail.com>
> > > >>>>>>> wrote:
> > > >>>>>>>>>>
> > > >>>>>>>>>>> Stephan, thanks a lot for explaining the file connector.
> > > >>> that
> > > >>>>>> makes
> > > >>>>>>>>>> sense.
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> I was asking because we were trying to reuse some of the
> > > >>>>>>>>> implementations
> > > >>>>>>>>>> in
> > > >>>>>>>>>>> the file source for Iceberg source. Flink Iceberg source
> > > >>> lives
> > > >>>>> in
> > > >>>>>>> the
> > > >>>>>>>>>>> Iceberg repo, which is not possible to code against the
> > > >>> master
> > > >>>>>>> branch
> > > >>>>>>>>> of
> > > >>>>>>>>>>> the Flink code.
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> On Mon, Nov 2, 2020 at 3:31 AM Stephan Ewen <
> > > >>> sewen@apache.org
> > > >>>>>
> > > >>>>>>> wrote:
> > > >>>>>>>>>>>
> > > >>>>>>>>>>>> Hi Steven!
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> So far there are no plans to pick back the file system
> > > >>>>> connector
> > > >>>>>>>>> code.
> > > >>>>>>>>>>> This
> > > >>>>>>>>>>>> is still evolving and not finalized for 1.12, so I don't
> > > >>>> feel
> > > >>>>> it
> > > >>>>>>> is
> > > >>>>>>>>> a
> > > >>>>>>>>>>> good
> > > >>>>>>>>>>>> candidate to be backported.
> > > >>>>>>>>>>>> However, with the base connector changes backported, you
> > > >>>>> should
> > > >>>>>> be
> > > >>>>>>>>> able
> > > >>>>>>>>>>> to
> > > >>>>>>>>>>>> run the file connector code from master against 1.11.3.
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> The collect() utils can be picked back, I see no issue
> > > >>> with
> > > >>>>> that
> > > >>>>>>>>> (it is
> > > >>>>>>>>>>>> isolated utilities).
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> Best,
> > > >>>>>>>>>>>> Stephan
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> On Mon, Nov 2, 2020 at 3:02 AM Steven Wu <
> > > >>>>> stevenz3wu@gmail.com>
> > > >>>>>>>>> wrote:
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>>> Basically, it would be great to get the latest code in
> > > >>> the
> > > >>>>>>>>>>>>> flink-connector-files (FLIP-27).
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>> On Sat, Oct 31, 2020 at 9:57 AM Steven Wu <
> > > >>>>>> stevenz3wu@gmail.com
> > > >>>>>>>>
> > > >>>>>>>>>>> wrote:
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>>> Stephan, it will be great if we can also backport
> > > >> the
> > > >>>>>>>>>> DataStreamUtils
> > > >>>>>>>>>>>>>> related commits that help with collecting output
> > > >> from
> > > >>>>>>> unbounded
> > > >>>>>>>>>>>> streams.
> > > >>>>>>>>>>>>>> e.g.
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>>
> > > >>>>>>>
> > > >>>>>>
> > > >>>>>
> > > >>>>
> > > >>>
> > > >>
> > >
> >
> https://github.com/apache/flink/commit/09a7a66b7313fea64817fe960a8da1265b428efc
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>> I tried to copy and paste the code to unblock
> > > >> myself.
> > > >>>> but
> > > >>>>> it
> > > >>>>>>>>>> quickly
> > > >>>>>>>>>>>> got
> > > >>>>>>>>>>>>>> into the rabbit hole of more and more code.
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>> On Fri, Oct 30, 2020 at 11:02 AM Stephan Ewen <
> > > >>>>>>> sewen@apache.org
> > > >>>>>>>>>>
> > > >>>>>>>>>>>> wrote:
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> I have started with backporting the source API
> > > >>> changes.
> > > >>>>>> Some
> > > >>>>>>>>> minor
> > > >>>>>>>>>>>>>>> conflicts to solve, will need a bit more to finish
> > > >>>> this.
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> On Fri, Oct 30, 2020 at 7:25 AM Tzu-Li (Gordon)
> > > >> Tai <
> > > >>>>>>>>>>>>> tzulitai@apache.org>
> > > >>>>>>>>>>>>>>> wrote:
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>> @Stephan Ewen <se...@apache.org>
> > > >>>>>>>>>>>>>>>> Are there already plans or ongoing efforts for
> > > >>>>>> backporting
> > > >>>>>>>>> the
> > > >>>>>>>>>>> list
> > > >>>>>>>>>>>> of
> > > >>>>>>>>>>>>>>>> FLIP-27 changes that you posted?
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>> On Thu, Oct 29, 2020 at 7:08 PM Xintong Song <
> > > >>>>>>>>>>> tonysong820@gmail.com
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>> wrote:
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>> Hi folks,
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>> Just to provide some updates concerning the
> > > >> status
> > > >>>> on
> > > >>>>>> the
> > > >>>>>>>>>>>>>>>>> test instabilities.
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>> Currently, we have 30 unresolved tickets labeled
> > > >>>> with
> > > >>>>>>>>> `Affects
> > > >>>>>>>>>>>>> Version`
> > > >>>>>>>>>>>>>>>>> 1.11.x.
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>>
> > > >>>>>>>
> > > >>>>>>
> > > >>>>>
> > > >>>>
> > > >>>
> > > >>
> > >
> >
> https://issues.apache.org/jira/browse/FLINK-19775?filter=12348580&jql=project%20%3D%20FLINK%20AND%20resolution%20%3D%20Unresolved%20AND%20affectedVersion%20in%20(1.11.0%2C%201.11.1%2C%201.11.2%2C%201.11.3)%20AND%20labels%20%3D%20test-stability%20ORDER%20BY%20created%20DESC
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>> Among the 30 tickets, 11 of them are:
> > > >>>>>>>>>>>>>>>>> - Have occured in the recent 3 months
> > > >>>>>>>>>>>>>>>>> - Not confirmed to be pure testability issues
> > > >>>>>>>>>>>>>>>>> - Not confirmed to be rare condition cases
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>> It would be nice if someone familiar with these
> > > >>>>>> components
> > > >>>>>>>>> can
> > > >>>>>>>>>>>> take a
> > > >>>>>>>>>>>>>>> look
> > > >>>>>>>>>>>>>>>>> into these issues.
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>> -
> > > >>> https://issues.apache.org/jira/browse/FLINK-17159
> > > >>>>>> (ES6)
> > > >>>>>>>>>>>>>>>>> -
> > > >>> https://issues.apache.org/jira/browse/FLINK-17912
> > > >>>>>>> (Kafka)
> > > >>>>>>>>>>>>>>>>> -
> > > >>> https://issues.apache.org/jira/browse/FLINK-17949
> > > >>>>>>> (Kafka)
> > > >>>>>>>>>>>>>>>>> ⁃
> > > >>> https://issues.apache.org/jira/browse/FLINK-18444
> > > >>>>>>> (Kafka)
> > > >>>>>>>>>>>>>>>>> -
> > > >>> https://issues.apache.org/jira/browse/FLINK-18634
> > > >>>>>>> (Kafka)
> > > >>>>>>>>>>>>>>>>> -
> > > >>> https://issues.apache.org/jira/browse/FLINK-18648
> > > >>>>>>> (Kafka)
> > > >>>>>>>>>>>>>>>>> -
> > > >>> https://issues.apache.org/jira/browse/FLINK-18807
> > > >>>>>>> (Kafka)
> > > >>>>>>>>>>>>>>>>> -
> > > >>> https://issues.apache.org/jira/browse/FLINK-19369
> > > >>>>>>>>>>>> (BlobClientTest)
> > > >>>>>>>>>>>>>>>>> -
> > > >>> https://issues.apache.org/jira/browse/FLINK-19436
> > > >>>>>>> (TPCDS)
> > > >>>>>>>>>>>>>>>>> -
> > > >>> https://issues.apache.org/jira/browse/FLINK-19690
> > > >>>>>>>>>>>> (Format/Parquet)
> > > >>>>>>>>>>>>>>>>> -
> > > >>> https://issues.apache.org/jira/browse/FLINK-19775
> > > >>>>>>>>>>>>>>>>> (SystemProcessingTimeServiceTest)
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>> Thank you~
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>> Xintong Song
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>> On Thu, Oct 29, 2020 at 10:21 AM Jingsong Li <
> > > >>>>>>>>>>>> jingsonglee0@gmail.com
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>> wrote:
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>> +1 to backport the FLIP-27 adjustments to
> > > >>> 1.11.x.
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>> If possible, that would be great. Many people
> > > >>> are
> > > >>>>>>> looking
> > > >>>>>>>>>>> forward
> > > >>>>>>>>>>>>> to
> > > >>>>>>>>>>>>>>> the
> > > >>>>>>>>>>>>>>>>>> FLIP-27 interface, but they don't want to take
> > > >>> the
> > > >>>>>> risk
> > > >>>>>>> to
> > > >>>>>>>>>>>> upgrade
> > > >>>>>>>>>>>>> to
> > > >>>>>>>>>>>>>>>>> 1.12
> > > >>>>>>>>>>>>>>>>>> (And wait 1.12). After all, 1.11 is a
> > > >> relatively
> > > >>>>>> stable
> > > >>>>>>>>>>> version.
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>> Best,
> > > >>>>>>>>>>>>>>>>>> Jingsong
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>> On Thu, Oct 29, 2020 at 1:24 AM Stephan Ewen <
> > > >>>>>>>>>> sewen@apache.org
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> wrote:
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>> Thanks for starting this.
> > > >>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>> +1 form my side to backport the FLIP-27
> > > >>>>> adjustments
> > > >>>>>> to
> > > >>>>>>>>>>> 1.11.x.
> > > >>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>> There were quite a few changes, and I think
> > > >> we
> > > >>>>> need
> > > >>>>>> to
> > > >>>>>>>>>>>>> cherry-pick
> > > >>>>>>>>>>>>>>>>> them
> > > >>>>>>>>>>>>>>>>>> all
> > > >>>>>>>>>>>>>>>>>>> to not get some inconsistent mix of changes
> > > >>> and
> > > >>>>> many
> > > >>>>>>>>> merge
> > > >>>>>>>>>>>>>>> conflicts.
> > > >>>>>>>>>>>>>>>>>>> I made a list below of what we need to add
> > > >> to
> > > >>>>>>>>>> "release-1.11".
> > > >>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>> *#### Core Source API Changes to backport
> > > >> (in
> > > >>>>>> REVERSE
> > > >>>>>>>>>> order)*
> > > >>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>  (Use: "git log
> > > >>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>
> > > >>>>> flink-core/src/main/java/org/apache/flink/api/connector/source")
> > > >>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>> commit
> > > >>> 162c072e9265a7b6dd9d6f5459eb7974091c4c4e
> > > >>>>>>>>>>>>>>>>>>> [FLINK-19492][core] Consolidate Source
> > > >> Events
> > > >>>>>> between
> > > >>>>>>>>>> Source
> > > >>>>>>>>>>>> API
> > > >>>>>>>>>>>>>>> and
> > > >>>>>>>>>>>>>>>>>> Split
> > > >>>>>>>>>>>>>>>>>>> Reader API
> > > >>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>> commit
> > > >>> 428fbbad85379f71a9fbf1a1ab3e3dcfa9006436
> > > >>>>>>>>>>>>>>>>>>> [FLINK-19205][core] Add access to
> > > >>> configuration
> > > >>>>> and
> > > >>>>>>>>>> hostname
> > > >>>>>>>>>>> in
> > > >>>>>>>>>>>>> the
> > > >>>>>>>>>>>>>>>>>>> SourceReaderContext
> > > >>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>> commit
> > > >>> 723e1790744ddc78d19c7c978442af1383f38d33
> > > >>>>>>>>>>>>>>>>>>> [hotfix][core] Add to Source Enumerator
> > > >>>>> convenience
> > > >>>>>>>>> methods
> > > >>>>>>>>>>> to
> > > >>>>>>>>>>>>>>> assign
> > > >>>>>>>>>>>>>>>>>>> single split
> > > >>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>> *#### Connector Base Changes to Backport (in
> > > >>>>> REVERSE
> > > >>>>>>>>>> order)*
> > > >>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>  (Use: "git log
> > > >>>>>>> flink-connectors/flink-connector-base")
> > > >>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>> commit
> > > >>> 401f56fe9d6b0271260edf9787cdcbfe4d03874d
> > > >>>>>>>>>>>>>>>>>>> [FLINK-19427][FLINK-19489][tests] Fix test
> > > >>>>>> conditions
> > > >>>>>>>>> for
> > > >>>>>>>>>>>>>>>>>>>
> > > >>>>>>> 'SplitFetcherTest.testNotifiesWhenGoingIdleConcurrent()'
> > > >>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>> commit
> > > >>> 68c5c2ff779d82a1ff81ffaf60d8a1b283797db1
> > > >>>>>>>>>>>>>>>>>>> [FLINK-19448][connector base] Explicitly
> > > >> check
> > > >>>> for
> > > >>>>>>>>>>> un-expected
> > > >>>>>>>>>>>>>>>>> condition
> > > >>>>>>>>>>>>>>>>>>> that would leave an inconsistent state
> > > >>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>> commit
> > > >>> 162c072e9265a7b6dd9d6f5459eb7974091c4c4e
> > > >>>>>>>>>>>>>>>>>>> [FLINK-19492][core] Consolidate Source
> > > >> Events
> > > >>>>>> between
> > > >>>>>>>>>> Source
> > > >>>>>>>>>>>> API
> > > >>>>>>>>>>>>>>> and
> > > >>>>>>>>>>>>>>>>>> Split
> > > >>>>>>>>>>>>>>>>>>> Reader API
> > > >>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>> commit
> > > >>> c1ca7a4c7c21ec8868c14cdddd43c559625b794c
> > > >>>>>>>>>>>>>>>>>>> [refactor][tests] Move some source test
> > > >> utils
> > > >>>> from
> > > >>>>>>>>>>>>>>>>> flink-connector-base
> > > >>>>>>>>>>>>>>>>>> to
> > > >>>>>>>>>>>>>>>>>>> flink-core
> > > >>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>> commit
> > > >>> ee5c4c211c35c70d28252363bbc8400453609977
> > > >>>>>>>>>>>>>>>>>>> [FLINK-19251][connectors] Avoid confusing
> > > >>> queue
> > > >>>>>>>>> handling in
> > > >>>>>>>>>>>>>>>>>>> "SplitReader.handleSplitsChanges()"
> > > >>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>> commit
> > > >>> 5abef56b2bf85bcac786f6b16b6899b6cced7176
> > > >>>>>>>>>>>>>>>>>>> [FLINK-19250][connectors] Fix error
> > > >>> propagation
> > > >>>> in
> > > >>>>>>>>>> connector
> > > >>>>>>>>>>>> base
> > > >>>>>>>>>>>>>>>>>>> (SplitFetcherManager).
> > > >>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>> commit
> > > >>> 8fcca837c55a9216595ee4c03038b52747098dbb
> > > >>>>>>>>>>>>>>>>>>> [hotfix][connectors] Improve JavaDocs for
> > > >>>>>>>>>>>>>>> SingleThreadFetcherManager
> > > >>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>> commit
> > > >>> 4700bb5dde3303cbe98882f6beb7379425717b01
> > > >>>>>>>>>>>>>>>>>>> [FLINK-19225][connectors] Various small
> > > >>>>> improvements
> > > >>>>>>> to
> > > >>>>>>>>>>>>>>>>> SourceReaderBase
> > > >>>>>>>>>>>>>>>>>>> (part 2)
> > > >>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>> commit
> > > >>> 12261c6b7ed6478a9b9f6a69cb58246b83cab9b7
> > > >>>>>>>>>>>>>>>>>>> [FLINK-17393][connectors] (follow-up) Wakeup
> > > >>> the
> > > >>>>>>>>>>> SplitFetchers
> > > >>>>>>>>>>>>> more
> > > >>>>>>>>>>>>>>>>>>> elegantly.
> > > >>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>> commit
> > > >>> c60aaff0249bfd6b5871b7f82e03efc487a54d6b
> > > >>>>>>>>>>>>>>>>>>> [hotfix][tests] Extend test coverage for
> > > >>>>>>>>>>>>>>> FutureCompletingBlockingQueue
> > > >>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>> commit
> > > >>> cef8a587d7fd2fe64cc644da5ed095d82e46f631
> > > >>>>>>>>>>>>>>>>>>> [FLINK-19245][connectors] Set default
> > > >> capacity
> > > >>>> for
> > > >>>>>>>>>>>>>>>>>>> FutureCompletingBlockingQueue.
> > > >>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>> commit
> > > >>> 4ea95782b4c6a2538153d4d16ad3f4839c7de0fb
> > > >>>>>>>>>>>>>>>>>>> [FLINK-19223][connectors] Simplify
> > > >>> Availability
> > > >>>>>> Future
> > > >>>>>>>>>> Model
> > > >>>>>>>>>>> in
> > > >>>>>>>>>>>>>>> Base
> > > >>>>>>>>>>>>>>>>>>> Connector
> > > >>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>> commit
> > > >>> 511857049ba30c8ff0ee56da551fa4a479dc583e
> > > >>>>>>>>>>>>>>>>>>> [FLINK-18128][connectors] Ensure idle split
> > > >>>>> fetchers
> > > >>>>>>>>> lead
> > > >>>>>>>>>> to
> > > >>>>>>>>>>>>>>>>> availability
> > > >>>>>>>>>>>>>>>>>>> notifications.
> > > >>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>> commit
> > > >>> a8206467af0830dcb89623ea068b5ca3b3450c92
> > > >>>>>>>>>>>>>>>>>>> [refactor][core] Eagerly initialize the
> > > >>>> FetchTask
> > > >>>>> to
> > > >>>>>>>>>> support
> > > >>>>>>>>>>>>> proper
> > > >>>>>>>>>>>>>>>>> unit
> > > >>>>>>>>>>>>>>>>>>> testing
> > > >>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>> commit
> > > >>> 3b2f54bcb437f98e6137c904045cc51072b5c06b
> > > >>>>>>>>>>>>>>>>>>> [hotfix][tests] Move constants in
> > > >>>> SplitFetcherTest
> > > >>>>>>>>> relevant
> > > >>>>>>>>>>> to
> > > >>>>>>>>>>>>> only
> > > >>>>>>>>>>>>>>>>> one
> > > >>>>>>>>>>>>>>>>>>> test into test method
> > > >>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>> commit
> > > >>> d7625760a75a508bf05bcddc380bb4d62ee1743e
> > > >>>>>>>>>>>>>>>>>>> [FLINK-19225][connectors] Various small
> > > >>>>> improvements
> > > >>>>>>> to
> > > >>>>>>>>>>>>>>>>> SourceReaderBase
> > > >>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>> commit
> > > >>> a5b0d3297748c1be47ad579a88f24df2255a8df1
> > > >>>>>>>>>>>>>>>>>>> [FLINK-17393][connectors] Wakeup the
> > > >>>> SplitFetchers
> > > >>>>>>> more
> > > >>>>>>>>>>>>> elegantly.
> > > >>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>> commit
> > > >>> f42a3ebc3e81a034b7221a803c153636fef34903
> > > >>>>>>>>>>>>>>>>>>> [FLINK-18680][connectors] Make connector
> > > >> base
> > > >>>>>>>>>>>> RecordsWithSplitIds
> > > >>>>>>>>>>>>>>> more
> > > >>>>>>>>>>>>>>>>>>> lightweight.
> > > >>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>> commit
> > > >>> e3d273de822b085183d09b275a445879ff94b350
> > > >>>>>>>>>>>>>>>>>>> [FLINK-19162][connectors] Add 'recycle()' to
> > > >>> the
> > > >>>>>>>>>>>>>>> RecordsWithSplitIds
> > > >>>>>>>>>>>>>>>>> to
> > > >>>>>>>>>>>>>>>>>>> support reuse of heavy objects.
> > > >>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>> commit
> > > >>> 8ebc464c2520453a70001cd712abc8dee6ee89e0
> > > >>>>>>>>>>>>>>>>>>> [hotfix][testing] Add a set of
> > > >> parameterizable
> > > >>>>>> testing
> > > >>>>>>>>>> mocks
> > > >>>>>>>>>>>> for
> > > >>>>>>>>>>>>>>> the
> > > >>>>>>>>>>>>>>>>>> Split
> > > >>>>>>>>>>>>>>>>>>> Reader API
> > > >>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>> commit
> > > >>> 930a07438be1185388d7150640f294dfe2a5d378
> > > >>>>>>>>>>>>>>>>>>> [hotfix][connectors] Add RequestSplitEvent
> > > >> to
> > > >>>>>>>>>>>>>>> 'flink-connector-base'
> > > >>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>> commit
> > > >>> e945ce8a933bc378844782f784ca473c767ca159
> > > >>>>>>>>>>>>>>>>>>> [refactor][connectors] Backport of the
> > > >>>>>> connector-base
> > > >>>>>>>>>>> exception
> > > >>>>>>>>>>>>>>>>> handling
> > > >>>>>>>>>>>>>>>>>>> from the Kafka Connector Pull Request
> > > >>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>> On Wed, Oct 28, 2020 at 5:49 PM Steven Wu <
> > > >>>>>>>>>>>> stevenz3wu@gmail.com>
> > > >>>>>>>>>>>>>>>>> wrote:
> > > >>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>> I would love to see this FLIP-27 source
> > > >>>>> interface
> > > >>>>>>>>>>> improvement
> > > >>>>>>>>>>>>> [1]
> > > >>>>>>>>>>>>>>>>> made
> > > >>>>>>>>>>>>>>>>>> to
> > > >>>>>>>>>>>>>>>>>>>> 1.11.3.
> > > >>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>> [1]
> > > >>>>>>> https://issues.apache.org/jira/browse/FLINK-19698
> > > >>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>> On Wed, Oct 28, 2020 at 12:32 AM Tzu-Li
> > > >>>> (Gordon)
> > > >>>>>>> Tai <
> > > >>>>>>>>>>>>>>>>>>> tzulitai@apache.org>
> > > >>>>>>>>>>>>>>>>>>>> wrote:
> > > >>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>> Thanks for the replies so far!
> > > >>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>> Just to provide a brief update on the
> > > >>> status
> > > >>>>> of
> > > >>>>>>>>>> blockers
> > > >>>>>>>>>>>> for
> > > >>>>>>>>>>>>>>>>> 1.11.3
> > > >>>>>>>>>>>>>>>>>> so
> > > >>>>>>>>>>>>>>>>>>>> far:
> > > >>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>> *PR opened, pending reviewer*-
> > > >>> [FLINK-19717]
> > > >>>>>>>>>>>>>>>>>> SourceReaderBase.pollNext
> > > >>>>>>>>>>>>>>>>>>>> may
> > > >>>>>>>>>>>>>>>>>>>>> return END_OF_INPUT if SplitReader.fetch
> > > >>>>> throws
> > > >>>>>> (
> > > >>>>>>>>>>>>>>>>>>>>>
> > > >>> https://github.com/apache/flink/pull/13776)
> > > >>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>> *PR opened, reviewed + close to being
> > > >>>> merged*
> > > >>>>>>>>>>>>>>>>>>>>> - [FLINK-19741] Timer service should
> > > >> skip
> > > >>>>>>> restoring
> > > >>>>>>>>>> from
> > > >>>>>>>>>>>> raw
> > > >>>>>>>>>>>>>>> keyed
> > > >>>>>>>>>>>>>>>>>>> stream
> > > >>>>>>>>>>>>>>>>>>>>> if it isn't the writer (
> > > >>>>>>>>>>>>>>>>> https://github.com/apache/flink/pull/13761)
> > > >>>>>>>>>>>>>>>>>>>>> - [FLINK-19748] Raw keyed stream key
> > > >> group
> > > >>>>>>> iterator
> > > >>>>>>>>>>> should
> > > >>>>>>>>>>>> be
> > > >>>>>>>>>>>>>>>>>> skipping
> > > >>>>>>>>>>>>>>>>>>>>> unwritten key groups (
> > > >>>>>>>>>>>>>>> https://github.com/apache/flink/pull/13772)
> > > >>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>> *Merged*
> > > >>>>>>>>>>>>>>>>>>>>> - [FLINK-19154] Application mode deletes
> > > >>> HA
> > > >>>>> data
> > > >>>>>>> in
> > > >>>>>>>>>> case
> > > >>>>>>>>>>> of
> > > >>>>>>>>>>>>>>>>> suspended
> > > >>>>>>>>>>>>>>>>>>>>> ZooKeeper connection
> > > >>>>>>>>>>>>>>>>>>>>> - [FLINK-19569] Upgrade ICU4J to 67.1+
> > > >>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>> Right now as it seems, progress is
> > > >> mainly
> > > >>>>>> blocked
> > > >>>>>>>>> on a
> > > >>>>>>>>>>>>> reviewer
> > > >>>>>>>>>>>>>>>>> for
> > > >>>>>>>>>>>>>>>>>>>>> FLINK-19717.
> > > >>>>>>>>>>>>>>>>>>>>> Meanwhile, Xintong is keeping an eye on
> > > >>> test
> > > >>>>>>>>>>> instabilities
> > > >>>>>>>>>>>>> [1]
> > > >>>>>>>>>>>>>>> to
> > > >>>>>>>>>>>>>>>>> see
> > > >>>>>>>>>>>>>>>>>>> if
> > > >>>>>>>>>>>>>>>>>>>>> there are any fixes that should be
> > > >> applied
> > > >>>> to
> > > >>>>>>>>>>>> `release-1.11`.
> > > >>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>> This is also a reminder, that if there
> > > >> are
> > > >>>>> other
> > > >>>>>>>>>> blockers
> > > >>>>>>>>>>>>> that
> > > >>>>>>>>>>>>>>> we
> > > >>>>>>>>>>>>>>>>>> need
> > > >>>>>>>>>>>>>>>>>>> to
> > > >>>>>>>>>>>>>>>>>>>>> be aware of, or a need to re-establish
> > > >>>>> estimated
> > > >>>>>>>>> time
> > > >>>>>>>>>> for
> > > >>>>>>>>>>>>>>> getting
> > > >>>>>>>>>>>>>>>>>> fixes
> > > >>>>>>>>>>>>>>>>>>>> in
> > > >>>>>>>>>>>>>>>>>>>>> and delay the RC for 1.11.3, please do
> > > >> let
> > > >>>> us
> > > >>>>>>> know!
> > > >>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>> Cheers,
> > > >>>>>>>>>>>>>>>>>>>>> Gordon
> > > >>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>> [1]
> > > >>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>>
> > > >>>>>>>
> > > >>>>>>
> > > >>>>>
> > > >>>>
> > > >>>
> > > >>
> > >
> >
> https://issues.apache.org/jira/issues/?jql=project%20%3D%20FLINK%20AND%20status%20in%20(Open%2C%20%22In%20Progress%22%2C%20Reopened)%20AND%20affectedVersion%20in%20(1.11.0%2C%201.11.1%2C%201.11.2)%20AND%20labels%20%3D%20test-stability
> > > >>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>> On Mon, Oct 26, 2020 at 9:43 PM Kostas
> > > >>>>> Kloudas <
> > > >>>>>>>>>>>>>>>>> kkloudas@apache.org>
> > > >>>>>>>>>>>>>>>>>>>>> wrote:
> > > >>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>> +1 for releasing Flink 1.11.3 as it
> > > >>>>> contains a
> > > >>>>>>>>> number
> > > >>>>>>>>>>> of
> > > >>>>>>>>>>>>>>>>> important
> > > >>>>>>>>>>>>>>>>>>>>>> fixes and thanks Gordon and Xintong
> > > >> for
> > > >>>>>>>>> volunteering.
> > > >>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>> Cheers,
> > > >>>>>>>>>>>>>>>>>>>>>> Kostas
> > > >>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>> On Mon, Oct 26, 2020 at 4:37 AM Yu Li
> > > >> <
> > > >>>>>>>>>>> carp84@gmail.com>
> > > >>>>>>>>>>>>>>> wrote:
> > > >>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>> +1 for releasing Flink 1.11.3, and
> > > >>>> thanks
> > > >>>>>>> Gordon
> > > >>>>>>>>>> and
> > > >>>>>>>>>>>>>>> Xintong
> > > >>>>>>>>>>>>>>>>> for
> > > >>>>>>>>>>>>>>>>>>>>>>> volunteering as our release
> > > >> managers.
> > > >>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>> Best Regards,
> > > >>>>>>>>>>>>>>>>>>>>>>> Yu
> > > >>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>> On Mon, 26 Oct 2020 at 09:45,
> > > >> Xintong
> > > >>>>> Song <
> > > >>>>>>>>>>>>>>>>>> tonysong820@gmail.com>
> > > >>>>>>>>>>>>>>>>>>>>>> wrote:
> > > >>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>> Thanks Gordan for starting this
> > > >>>>>> discussion.
> > > >>>>>>>>>>>>>>>>>>>>>>>> My pleasure to help with the
> > > >> release
> > > >>>>>>> process.
> > > >>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>> Thank you~
> > > >>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>> Xintong Song
> > > >>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>> On Fri, Oct 23, 2020 at 11:29 PM
> > > >>> Till
> > > >>>>>>>>> Rohrmann <
> > > >>>>>>>>>>>>>>>>>>>> trohrmann@apache.org
> > > >>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>> wrote:
> > > >>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>> Thanks for starting this
> > > >>> discussion
> > > >>>>>>> Gordon.
> > > >>>>>>>>>> There
> > > >>>>>>>>>>>> are
> > > >>>>>>>>>>>>>>> over
> > > >>>>>>>>>>>>>>>>>> 100
> > > >>>>>>>>>>>>>>>>>>>>> issues
> > > >>>>>>>>>>>>>>>>>>>>>>>>> which are fixed for 1.11.3.
> > > >> Hence
> > > >>> +1
> > > >>>>>> for a
> > > >>>>>>>>>>> soonish
> > > >>>>>>>>>>>>>>> 1.11.3
> > > >>>>>>>>>>>>>>>>>>>> release.
> > > >>>>>>>>>>>>>>>>>>>>>> Thanks
> > > >>>>>>>>>>>>>>>>>>>>>>>>> for volunteering as our release
> > > >>>>> managers
> > > >>>>>>>>> Gordon
> > > >>>>>>>>>>> and
> > > >>>>>>>>>>>>>>>>> Xintong!
> > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>> Cheers,
> > > >>>>>>>>>>>>>>>>>>>>>>>>> Till
> > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>> On Fri, Oct 23, 2020 at 5:02 PM
> > > >>>> Tzu-Li
> > > >>>>>>>>> (Gordon)
> > > >>>>>>>>>>>> Tai <
> > > >>>>>>>>>>>>>>>>>>>>>> tzulitai@apache.org
> > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>> wrote:
> > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> Hi,
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> Xintong and I would like to
> > > >>> start a
> > > >>>>>>>>> discussion
> > > >>>>>>>>>>> for
> > > >>>>>>>>>>>>>>>>> releasing
> > > >>>>>>>>>>>>>>>>>>>> Flink
> > > >>>>>>>>>>>>>>>>>>>>>>>> 1.11.3
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> soon.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> It seems like we already have a
> > > >>> few
> > > >>>>>>>>> pressing
> > > >>>>>>>>>>>> issues
> > > >>>>>>>>>>>>>>> that
> > > >>>>>>>>>>>>>>>>>> needs
> > > >>>>>>>>>>>>>>>>>>>> to
> > > >>>>>>>>>>>>>>>>>>>>> be
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> included in a new hotfix
> > > >> release:
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>   - Heap-based timers’ restore
> > > >>>>>> behaviour
> > > >>>>>>>>> is
> > > >>>>>>>>>>>>> causing a
> > > >>>>>>>>>>>>>>>>>>> critical
> > > >>>>>>>>>>>>>>>>>>>>>> recovery
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>   issue for StateFun [1] [2]
> > > >>> [3].
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>   - There are several
> > > >> robustness
> > > >>>>>> issues
> > > >>>>>>>>> for
> > > >>>>>>>>>> the
> > > >>>>>>>>>>>>>>> FLIP-27
> > > >>>>>>>>>>>>>>>>> new
> > > >>>>>>>>>>>>>>>>>>>>> source
> > > >>>>>>>>>>>>>>>>>>>>>> API,
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>   such as [4]. We already have
> > > >>>> some
> > > >>>>>>> users
> > > >>>>>>>>>> using
> > > >>>>>>>>>>>> the
> > > >>>>>>>>>>>>>>>>> FLIP-27
> > > >>>>>>>>>>>>>>>>>>> API
> > > >>>>>>>>>>>>>>>>>>>>>> with
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> 1.11.x,
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>   so it would be important to
> > > >>> get
> > > >>>>>> those
> > > >>>>>>>>> fixes
> > > >>>>>>>>>>> in
> > > >>>>>>>>>>>>> for
> > > >>>>>>>>>>>>>>>>> 1.11.x
> > > >>>>>>>>>>>>>>>>>>> as
> > > >>>>>>>>>>>>>>>>>>>>>> well.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> Apart from the issues that are
> > > >>>>> already
> > > >>>>>>>>> marked
> > > >>>>>>>>>> as
> > > >>>>>>>>>>>>>>> blocker
> > > >>>>>>>>>>>>>>>>> for
> > > >>>>>>>>>>>>>>>>>>>>> 1.11.3
> > > >>>>>>>>>>>>>>>>>>>>>> in
> > > >>>>>>>>>>>>>>>>>>>>>>>> our
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> JIRA [5], please let us know in
> > > >>>> this
> > > >>>>>>>>> thread if
> > > >>>>>>>>>>>> there
> > > >>>>>>>>>>>>>>> is
> > > >>>>>>>>>>>>>>>>>>> already
> > > >>>>>>>>>>>>>>>>>>>>>> ongoing
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> work for other important fixes
> > > >>> that
> > > >>>>> we
> > > >>>>>>>>> should
> > > >>>>>>>>>>> try
> > > >>>>>>>>>>>> to
> > > >>>>>>>>>>>>>>>>>> include.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> Xintong and I would like to
> > > >>>> volunteer
> > > >>>>>> for
> > > >>>>>>>>>>> managing
> > > >>>>>>>>>>>>>>> this
> > > >>>>>>>>>>>>>>>>>>> release,
> > > >>>>>>>>>>>>>>>>>>>>> and
> > > >>>>>>>>>>>>>>>>>>>>>>>> will
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> try to communicate the priority
> > > >>> of
> > > >>>>>>> pending
> > > >>>>>>>>>>>> blockers
> > > >>>>>>>>>>>>>>> over
> > > >>>>>>>>>>>>>>>>> the
> > > >>>>>>>>>>>>>>>>>>>> next
> > > >>>>>>>>>>>>>>>>>>>>>> few
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> days.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> Since the aforementioned issues
> > > >>> are
> > > >>>>>> quite
> > > >>>>>>>>>>>> critical,
> > > >>>>>>>>>>>>>>> we’d
> > > >>>>>>>>>>>>>>>>>> like
> > > >>>>>>>>>>>>>>>>>>> to
> > > >>>>>>>>>>>>>>>>>>>>> aim
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> for a *feature
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> freeze by the end of next week
> > > >>>> (Oct.
> > > >>>>>>> 30th)*
> > > >>>>>>>>>> and
> > > >>>>>>>>>>>>> start
> > > >>>>>>>>>>>>>>> the
> > > >>>>>>>>>>>>>>>>>>>> release
> > > >>>>>>>>>>>>>>>>>>>>>> voting
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> process the week after.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> If that is too short of a
> > > >> notice
> > > >>>> and
> > > >>>>>> you
> > > >>>>>>>>> might
> > > >>>>>>>>>>>> need
> > > >>>>>>>>>>>>>>> more
> > > >>>>>>>>>>>>>>>>>> time,
> > > >>>>>>>>>>>>>>>>>>>>>> please
> > > >>>>>>>>>>>>>>>>>>>>>>>> let
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> us know!
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> Cheers,
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> Gordon
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> [1]
> > > >>>>>>>>>>>>> https://issues.apache.org/jira/browse/FLINK-19692
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> [2]
> > > >>>>>>>>>>>>> https://issues.apache.org/jira/browse/FLINK-19741
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> [3]
> > > >>>>>>>>>>>>> https://issues.apache.org/jira/browse/FLINK-19748
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> [4]
> > > >>>>>>>>>>>>> https://issues.apache.org/jira/browse/FLINK-19717
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> [5]
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>>
> > > >>>>>>>
> > > >>>>>>
> > > >>>>>
> > > >>>>
> > > >>>
> > > >>
> > >
> >
> https://issues.apache.org/jira/issues/?jql=project%20%3D%20FLINK%20AND%20priority%20%3D%20Blocker%20AND%20fixVersion%20%3D%201.11.3
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>> --
> > > >>>>>>>>>>>>>>>>>> Best, Jingsong Lee
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>>
> > > >>>>>>>>
> > > >>>>>>>
> > > >>>>>>
> > > >>>>>
> > > >>>>
> > > >>>
> > > >>
> > >
> > >
> > >
> >
>

Re: [DISCUSS] Releasing Apache Flink 1.11.3

Posted by Khachatryan Roman <kh...@gmail.com>.
Hi,

I'd like FLINK-20079 [1] to be merged into 1.11 and included in 1.11.3.

[1] https://issues.apache.org/jira/browse/FLINK-20079

Regards,
Roman


On Tue, Nov 10, 2020 at 8:21 AM Xintong Song <to...@gmail.com> wrote:

> Thanks for the notice, Dian.
>
> Thank you~
>
> Xintong Song
>
>
>
> On Tue, Nov 10, 2020 at 10:19 AM Dian Fu <di...@gmail.com> wrote:
>
> > Hi Xintong,
> >
> > I want to bring one more issue to your attention [1]. The test case
> > UnalignedCheckpointCompatibilityITCase.test failed several times in the
> > last nightly test of release-1.11. We need to figure out if it's just an
> > instable test or caused by recent changes.
> >
> > [1] https://issues.apache.org/jira/browse/FLINK-20065
> >
> > > 在 2020年11月10日，上午9:24，Xintong Song <to...@gmail.com> 写道：
> > >
> > > Thanks for the replies.
> > >
> > > Thank you~
> > >
> > > Xintong Song
> > >
> > >
> > >
> > > On Tue, Nov 10, 2020 at 1:09 AM Becket Qin <be...@gmail.com>
> wrote:
> > >
> > >> Hi Xintong,
> > >>
> > >> Thanks for driving the release. Just want to sync up on the FLIP-27
> > >> backporting. Stephan and I are still trying to backport a bunch of
> > patches
> > >> of Source to 1.11.3. Including:
> > >>
> > >> [FLINK-19698][connector/common] Add a close() method to the
> SplitReader.
> > >> [FLINK-19717] SourceReaderBase.pollNext may return END_OF_INPUT if
> > >> SplitReader.fetch throws
> > >> [FLINK-19535] [connector/common] Avoid failing a job multiple times in
> > >> SourceCoordinator.
> > >> [FLINK-19265] [FLINK-20049][core] Source API final adjustments.
> > >>
> > >> and a few more fixes.
> > >>
> > >> We are currently trying to fix them in 1.12 first so it might take a
> > little
> > >> longer to backport them to 1.11.3. I think it will probably take us a
> > few
> > >> more days to finish the backport. So that would roughly be the end of
> > this
> > >> week.
> > >>
> > >> Thanks,
> > >>
> > >> Jiangjie (Becket) Qin
> > >>
> > >>
> > >>
> > >>
> > >> On Mon, Nov 9, 2020 at 9:57 PM Till Rohrmann <tr...@apache.org>
> > wrote:
> > >>
> > >>> Yes, I've downgraded FLINK-19816 to critical.
> > >>>
> > >>> Cheers,
> > >>> Till
> > >>>
> > >>> On Mon, Nov 9, 2020 at 10:19 AM Xintong Song <to...@gmail.com>
> > >>> wrote:
> > >>>
> > >>>> Thanks for the notice, Till.
> > >>>>
> > >>>> I just checked and found FLINK-20033 is already fixed. Shall we also
> > >>>> downgrade FLINK-19816 to `Critical`?
> > >>>>
> > >>>> Thank you~
> > >>>>
> > >>>> Xintong Song
> > >>>>
> > >>>>
> > >>>>
> > >>>> On Mon, Nov 9, 2020 at 4:42 PM Till Rohrmann <tr...@apache.org>
> > >>> wrote:
> > >>>>
> > >>>>> I would like to bring one more critical issue to your attention
> which
> > >>> is
> > >>>>> FLINK-20033 [1]. I believe that this issue is actually causing what
> > >> has
> > >>>>> been reported in FLINK-19816 [2]. I hope to have it fixed by the
> end
> > >> of
> > >>>>> today. Once FLINK-20033 is fixed, I think that we don't have to
> block
> > >>> the
> > >>>>> release on FLINK-19816.
> > >>>>>
> > >>>>> [1] https://issues.apache.org/jira/browse/FLINK-20033
> > >>>>> [2] https://issues.apache.org/jira/browse/FLINK-19816
> > >>>>>
> > >>>>> Cheers,
> > >>>>> Till
> > >>>>>
> > >>>>> On Mon, Nov 9, 2020 at 4:05 AM Xintong Song <tonysong820@gmail.com
> >
> > >>>> wrote:
> > >>>>>
> > >>>>>> Hi devs,
> > >>>>>>
> > >>>>>> I'd like to provide an update on the progress of preparing release
> > >>>>> 1.11.3.
> > >>>>>>
> > >>>>>> *Blockers*
> > >>>>>> We currently have 3 remaining blockers. (3 resolved and 1 emerged
> > >>>>> compared
> > >>>>>> to last week)
> > >>>>>>
> > >>>>>> - [FLINK-19698] Add close() method and onCheckpointComplete() to
> > >> the
> > >>>>>> Source.
> > >>>>>> The issue has been fixed on the master branch. It's currently
> > >> blocked
> > >>>> on
> > >>>>>> the FLIP-27 backportings to backport it to the 1.11 branch.
> > >>>>>>
> > >>>>>> - [FLINK-19717] SourceReaderBase.pollNext may return END_OF_INPUT
> > >> if
> > >>>>>> SplitReader.fetch throws
> > >>>>>> A PR has been opened and reviewed. From the discussions on the PR,
> > >> it
> > >>>>> looks
> > >>>>>> close to mergeable.
> > >>>>>>
> > >>>>>> - [FLINK-19816] Flink restored from a wrong checkpoint (a very old
> > >>> one
> > >>>>> and
> > >>>>>> not the last completed one)
> > >>>>>> This is a newly emerged blocker and Matthias is working on it.
> > >>>>>>
> > >>>>>> *Test Instabilities*
> > >>>>>> We currently have 27 test instabilities[1].
> > >>>>>> AFAIK, none of them are as serious as to block the 1.11.3 release.
> > >>>>>>
> > >>>>>> *FLIP-27 Backprotings*
> > >>>>>>
> > >>>>>> I noticed that there's no jira issues opened on the FLIP-27
> > >>> backporting
> > >>>>>> efforts, which is part of the major efforts planned for the 1.11.3
> > >>>>> release,
> > >>>>>> making it hard to track the progress.
> > >>>>>>
> > >>>>>>
> > >>>>>> @Stephan and @Becket, could you please share the updates on the
> > >>>>> backporting
> > >>>>>> efforts? How is the progress and when are the efforts expected to
> > >> be
> > >>>>>> finished? It would be appreciated and helpful if we can have a
> jira
> > >>>>> ticket
> > >>>>>> to track the backporing progress publicly.
> > >>>>>>
> > >>>>>>
> > >>>>>> Thank you~
> > >>>>>>
> > >>>>>> Xintong Song
> > >>>>>>
> > >>>>>>
> > >>>>>> [1]
> > >>>>>>
> > >>>>>>
> > >>>>>
> > >>>>
> > >>>
> > >>
> >
> https://issues.apache.org/jira/browse/FLINK-16908?filter=12348580&jql=project%20%3D%20FLINK%20AND%20resolution%20%3D%20Unresolved%20AND%20affectedVersion%20in%20(1.11.0%2C%201.11.1%2C%201.11.2%2C%201.11.3)%20AND%20labels%20%3D%20test-stability%20ORDER%20BY%20updated%20DESC%2C%20created%20DESC
> > >>>>>>
> > >>>>>> On Wed, Nov 4, 2020 at 3:58 PM Tzu-Li (Gordon) Tai <
> > >>>> tzulitai@apache.org>
> > >>>>>> wrote:
> > >>>>>>
> > >>>>>>> Thanks for the efforts so far with pushing for 1.11.3.
> > >>>>>>>
> > >>>>>>> I'd like to provide a mid-week update on how we're looking with
> > >> all
> > >>>> the
> > >>>>>>> pending blockers and backports:
> > >>>>>>>
> > >>>>>>> *Blockers**:*
> > >>>>>>>
> > >>>>>>> Currently all blockers either have an open PR, or have been
> > >> merged.
> > >>>>> I'll
> > >>>>>>> highlight below the blockers *that still require reviewing
> > >> efforts
> > >>>> for
> > >>>>>> them
> > >>>>>>> to move forward:*
> > >>>>>>>
> > >>>>>>> - [FLINK-19909] Flink application in attach mode could not
> > >>> terminate
> > >>>>> when
> > >>>>>>> the only job is canceled. PR:
> > >>>>> https://github.com/apache/flink/pull/13911
> > >>>>>>> - [FLINK-19717] SourceReaderBase.pollNext may return END_OF_INPUT
> > >>> if
> > >>>>>>> SplitReader.fetch throws. PR:
> > >>>>> https://github.com/apache/flink/pull/13776
> > >>>>>>>
> > >>>>>>> The above PRs currently have no reviews at all yet. They seem to
> > >>>>> already
> > >>>>>>> have designated reviewers.
> > >>>>>>>
> > >>>>>>>
> > >>>>>>> *Backports:*
> > >>>>>>> - FLIP-27 robustness improvement backports: Stephan is currently
> > >>>>> working
> > >>>>>> on
> > >>>>>>> backporting several FLIP-27 changes. There are no PRs yet for the
> > >>>>>>> backported changes.
> > >>>>>>> - DataStreamUtils.collect() refactorings backport: Steven Wu
> > >>>> mentioned
> > >>>>> to
> > >>>>>>> backport this, but AFAIK this isn't assigned to anyone yet.
> > >>>>>>>
> > >>>>>>> *ETAs*:
> > >>>>>>>
> > >>>>>>> I'd like to request ETAs for the remaining backports, to prevent
> > >> a
> > >>>>> creep
> > >>>>>> in
> > >>>>>>> the scope of this bugfix release.
> > >>>>>>>
> > >>>>>>> We already have Flink users that would benefit from fixes that
> > >> have
> > >>>>> been
> > >>>>>>> merged to release-1.11 already, so technically speaking the
> > >>> backports
> > >>>>>>> should be considered "nice-to-have" (to the best of my knowledge
> > >> of
> > >>>> the
> > >>>>>>> changes) and could potentially be moved to a follow-up 1.11.4.
> > >>>>>>> Most notably, the Stateful Functions project is already waiting
> > >> on
> > >>>>> Flink
> > >>>>>>> 1.11.3 to address critical recovery issues (please see the
> > >> StateFun
> > >>>>> 2.2.1
> > >>>>>>> release discussion thread [1]).
> > >>>>>>>
> > >>>>>>> @Stephan Ewen <se...@apache.org> @Becket Qin <
> > >> becket.qin@gmail.com
> > >>>>
> > >>>>>> could
> > >>>>>>> you provide an ETA for the FLIP-27 backports? It would help to
> > >> get
> > >>> a
> > >>>>>> better
> > >>>>>>> estimate to decide how we proceed here.
> > >>>>>>>
> > >>>>>>> Cheers,
> > >>>>>>> Gordon
> > >>>>>>>
> > >>>>>>> [1]
> > >>>>>>>
> > >>>>>>>
> > >>>>>>
> > >>>>>
> > >>>>
> > >>>
> > >>
> >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Releasing-StateFun-hotfix-version-2-2-1-td46239.html
> > >>>>>>>
> > >>>>>>> On Wed, Nov 4, 2020 at 3:16 PM Tzu-Li (Gordon) Tai <
> > >>>>> tzulitai@apache.org>
> > >>>>>>> wrote:
> > >>>>>>>
> > >>>>>>>>> The collect() utils can be picked back, I see no issue with
> > >>> that
> > >>>>> (it
> > >>>>>> is
> > >>>>>>>> isolated utilities).
> > >>>>>>>>
> > >>>>>>>> Just checking on all the requested backports mentioned in this
> > >>>>> thread,
> > >>>>>>> and
> > >>>>>>>> figuring out which ones seem to still be unassigned / open.
> > >>>>>>>>
> > >>>>>>>> Is someone working on backporting
> > >>>>>>>>
> > >>>>>>>
> > >>>>>>
> > >>>>>
> > >>>>
> > >>>
> > >>
> >
> https://github.com/apache/flink/commit/09a7a66b7313fea64817fe960a8da1265b428efc
> > >>>>>>>> to release-1.11 at the moment?
> > >>>>>>>>
> > >>>>>>>> On Wed, Nov 4, 2020 at 1:39 AM Steven Wu <stevenz3wu@gmail.com
> > >>>
> > >>>>> wrote:
> > >>>>>>>>
> > >>>>>>>>> @Stephan Ewen <se...@apache.org> yeah, we can do that. don't
> > >>>> worry
> > >>>>>>> about
> > >>>>>>>>> it. your earlier email had the perfect explanation on why file
> > >>>>> source
> > >>>>>>>>> shouldn't be backported.
> > >>>>>>>>>
> > >>>>>>>>> On Tue, Nov 3, 2020 at 3:37 AM Stephan Ewen <sewen@apache.org
> > >>>
> > >>>>> wrote:
> > >>>>>>>>>
> > >>>>>>>>>> @Steven would it be possible to initially copy some of the
> > >>> code
> > >>>>> into
> > >>>>>>> the
> > >>>>>>>>>> iceberg source and later replace it by a dependency on the
> > >>> Flink
> > >>>>>> file
> > >>>>>>>>>> source?
> > >>>>>>>>>>
> > >>>>>>>>>> On Mon, Nov 2, 2020 at 8:33 PM Steven Wu <
> > >>> stevenz3wu@gmail.com>
> > >>>>>>> wrote:
> > >>>>>>>>>>
> > >>>>>>>>>>> Stephan, thanks a lot for explaining the file connector.
> > >>> that
> > >>>>>> makes
> > >>>>>>>>>> sense.
> > >>>>>>>>>>>
> > >>>>>>>>>>> I was asking because we were trying to reuse some of the
> > >>>>>>>>> implementations
> > >>>>>>>>>> in
> > >>>>>>>>>>> the file source for Iceberg source. Flink Iceberg source
> > >>> lives
> > >>>>> in
> > >>>>>>> the
> > >>>>>>>>>>> Iceberg repo, which is not possible to code against the
> > >>> master
> > >>>>>>> branch
> > >>>>>>>>> of
> > >>>>>>>>>>> the Flink code.
> > >>>>>>>>>>>
> > >>>>>>>>>>> On Mon, Nov 2, 2020 at 3:31 AM Stephan Ewen <
> > >>> sewen@apache.org
> > >>>>>
> > >>>>>>> wrote:
> > >>>>>>>>>>>
> > >>>>>>>>>>>> Hi Steven!
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> So far there are no plans to pick back the file system
> > >>>>> connector
> > >>>>>>>>> code.
> > >>>>>>>>>>> This
> > >>>>>>>>>>>> is still evolving and not finalized for 1.12, so I don't
> > >>>> feel
> > >>>>> it
> > >>>>>>> is
> > >>>>>>>>> a
> > >>>>>>>>>>> good
> > >>>>>>>>>>>> candidate to be backported.
> > >>>>>>>>>>>> However, with the base connector changes backported, you
> > >>>>> should
> > >>>>>> be
> > >>>>>>>>> able
> > >>>>>>>>>>> to
> > >>>>>>>>>>>> run the file connector code from master against 1.11.3.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> The collect() utils can be picked back, I see no issue
> > >>> with
> > >>>>> that
> > >>>>>>>>> (it is
> > >>>>>>>>>>>> isolated utilities).
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Best,
> > >>>>>>>>>>>> Stephan
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> On Mon, Nov 2, 2020 at 3:02 AM Steven Wu <
> > >>>>> stevenz3wu@gmail.com>
> > >>>>>>>>> wrote:
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>> Basically, it would be great to get the latest code in
> > >>> the
> > >>>>>>>>>>>>> flink-connector-files (FLIP-27).
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> On Sat, Oct 31, 2020 at 9:57 AM Steven Wu <
> > >>>>>> stevenz3wu@gmail.com
> > >>>>>>>>
> > >>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>>> Stephan, it will be great if we can also backport
> > >> the
> > >>>>>>>>>> DataStreamUtils
> > >>>>>>>>>>>>>> related commits that help with collecting output
> > >> from
> > >>>>>>> unbounded
> > >>>>>>>>>>>> streams.
> > >>>>>>>>>>>>>> e.g.
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>
> > >>>>>>
> > >>>>>
> > >>>>
> > >>>
> > >>
> >
> https://github.com/apache/flink/commit/09a7a66b7313fea64817fe960a8da1265b428efc
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> I tried to copy and paste the code to unblock
> > >> myself.
> > >>>> but
> > >>>>> it
> > >>>>>>>>>> quickly
> > >>>>>>>>>>>> got
> > >>>>>>>>>>>>>> into the rabbit hole of more and more code.
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> On Fri, Oct 30, 2020 at 11:02 AM Stephan Ewen <
> > >>>>>>> sewen@apache.org
> > >>>>>>>>>>
> > >>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> I have started with backporting the source API
> > >>> changes.
> > >>>>>> Some
> > >>>>>>>>> minor
> > >>>>>>>>>>>>>>> conflicts to solve, will need a bit more to finish
> > >>>> this.
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> On Fri, Oct 30, 2020 at 7:25 AM Tzu-Li (Gordon)
> > >> Tai <
> > >>>>>>>>>>>>> tzulitai@apache.org>
> > >>>>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> @Stephan Ewen <se...@apache.org>
> > >>>>>>>>>>>>>>>> Are there already plans or ongoing efforts for
> > >>>>>> backporting
> > >>>>>>>>> the
> > >>>>>>>>>>> list
> > >>>>>>>>>>>> of
> > >>>>>>>>>>>>>>>> FLIP-27 changes that you posted?
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> On Thu, Oct 29, 2020 at 7:08 PM Xintong Song <
> > >>>>>>>>>>> tonysong820@gmail.com
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> Hi folks,
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> Just to provide some updates concerning the
> > >> status
> > >>>> on
> > >>>>>> the
> > >>>>>>>>>>>>>>>>> test instabilities.
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> Currently, we have 30 unresolved tickets labeled
> > >>>> with
> > >>>>>>>>> `Affects
> > >>>>>>>>>>>>> Version`
> > >>>>>>>>>>>>>>>>> 1.11.x.
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>
> > >>>>>>
> > >>>>>
> > >>>>
> > >>>
> > >>
> >
> https://issues.apache.org/jira/browse/FLINK-19775?filter=12348580&jql=project%20%3D%20FLINK%20AND%20resolution%20%3D%20Unresolved%20AND%20affectedVersion%20in%20(1.11.0%2C%201.11.1%2C%201.11.2%2C%201.11.3)%20AND%20labels%20%3D%20test-stability%20ORDER%20BY%20created%20DESC
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> Among the 30 tickets, 11 of them are:
> > >>>>>>>>>>>>>>>>> - Have occured in the recent 3 months
> > >>>>>>>>>>>>>>>>> - Not confirmed to be pure testability issues
> > >>>>>>>>>>>>>>>>> - Not confirmed to be rare condition cases
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> It would be nice if someone familiar with these
> > >>>>>> components
> > >>>>>>>>> can
> > >>>>>>>>>>>> take a
> > >>>>>>>>>>>>>>> look
> > >>>>>>>>>>>>>>>>> into these issues.
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> -
> > >>> https://issues.apache.org/jira/browse/FLINK-17159
> > >>>>>> (ES6)
> > >>>>>>>>>>>>>>>>> -
> > >>> https://issues.apache.org/jira/browse/FLINK-17912
> > >>>>>>> (Kafka)
> > >>>>>>>>>>>>>>>>> -
> > >>> https://issues.apache.org/jira/browse/FLINK-17949
> > >>>>>>> (Kafka)
> > >>>>>>>>>>>>>>>>> ⁃
> > >>> https://issues.apache.org/jira/browse/FLINK-18444
> > >>>>>>> (Kafka)
> > >>>>>>>>>>>>>>>>> -
> > >>> https://issues.apache.org/jira/browse/FLINK-18634
> > >>>>>>> (Kafka)
> > >>>>>>>>>>>>>>>>> -
> > >>> https://issues.apache.org/jira/browse/FLINK-18648
> > >>>>>>> (Kafka)
> > >>>>>>>>>>>>>>>>> -
> > >>> https://issues.apache.org/jira/browse/FLINK-18807
> > >>>>>>> (Kafka)
> > >>>>>>>>>>>>>>>>> -
> > >>> https://issues.apache.org/jira/browse/FLINK-19369
> > >>>>>>>>>>>> (BlobClientTest)
> > >>>>>>>>>>>>>>>>> -
> > >>> https://issues.apache.org/jira/browse/FLINK-19436
> > >>>>>>> (TPCDS)
> > >>>>>>>>>>>>>>>>> -
> > >>> https://issues.apache.org/jira/browse/FLINK-19690
> > >>>>>>>>>>>> (Format/Parquet)
> > >>>>>>>>>>>>>>>>> -
> > >>> https://issues.apache.org/jira/browse/FLINK-19775
> > >>>>>>>>>>>>>>>>> (SystemProcessingTimeServiceTest)
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> Thank you~
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> Xintong Song
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> On Thu, Oct 29, 2020 at 10:21 AM Jingsong Li <
> > >>>>>>>>>>>> jingsonglee0@gmail.com
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> +1 to backport the FLIP-27 adjustments to
> > >>> 1.11.x.
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> If possible, that would be great. Many people
> > >>> are
> > >>>>>>> looking
> > >>>>>>>>>>> forward
> > >>>>>>>>>>>>> to
> > >>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>> FLIP-27 interface, but they don't want to take
> > >>> the
> > >>>>>> risk
> > >>>>>>> to
> > >>>>>>>>>>>> upgrade
> > >>>>>>>>>>>>> to
> > >>>>>>>>>>>>>>>>> 1.12
> > >>>>>>>>>>>>>>>>>> (And wait 1.12). After all, 1.11 is a
> > >> relatively
> > >>>>>> stable
> > >>>>>>>>>>> version.
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> Best,
> > >>>>>>>>>>>>>>>>>> Jingsong
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> On Thu, Oct 29, 2020 at 1:24 AM Stephan Ewen <
> > >>>>>>>>>> sewen@apache.org
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> Thanks for starting this.
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> +1 form my side to backport the FLIP-27
> > >>>>> adjustments
> > >>>>>> to
> > >>>>>>>>>>> 1.11.x.
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> There were quite a few changes, and I think
> > >> we
> > >>>>> need
> > >>>>>> to
> > >>>>>>>>>>>>> cherry-pick
> > >>>>>>>>>>>>>>>>> them
> > >>>>>>>>>>>>>>>>>> all
> > >>>>>>>>>>>>>>>>>>> to not get some inconsistent mix of changes
> > >>> and
> > >>>>> many
> > >>>>>>>>> merge
> > >>>>>>>>>>>>>>> conflicts.
> > >>>>>>>>>>>>>>>>>>> I made a list below of what we need to add
> > >> to
> > >>>>>>>>>> "release-1.11".
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> *#### Core Source API Changes to backport
> > >> (in
> > >>>>>> REVERSE
> > >>>>>>>>>> order)*
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>  (Use: "git log
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>> flink-core/src/main/java/org/apache/flink/api/connector/source")
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> commit
> > >>> 162c072e9265a7b6dd9d6f5459eb7974091c4c4e
> > >>>>>>>>>>>>>>>>>>> [FLINK-19492][core] Consolidate Source
> > >> Events
> > >>>>>> between
> > >>>>>>>>>> Source
> > >>>>>>>>>>>> API
> > >>>>>>>>>>>>>>> and
> > >>>>>>>>>>>>>>>>>> Split
> > >>>>>>>>>>>>>>>>>>> Reader API
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> commit
> > >>> 428fbbad85379f71a9fbf1a1ab3e3dcfa9006436
> > >>>>>>>>>>>>>>>>>>> [FLINK-19205][core] Add access to
> > >>> configuration
> > >>>>> and
> > >>>>>>>>>> hostname
> > >>>>>>>>>>> in
> > >>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>> SourceReaderContext
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> commit
> > >>> 723e1790744ddc78d19c7c978442af1383f38d33
> > >>>>>>>>>>>>>>>>>>> [hotfix][core] Add to Source Enumerator
> > >>>>> convenience
> > >>>>>>>>> methods
> > >>>>>>>>>>> to
> > >>>>>>>>>>>>>>> assign
> > >>>>>>>>>>>>>>>>>>> single split
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> *#### Connector Base Changes to Backport (in
> > >>>>> REVERSE
> > >>>>>>>>>> order)*
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>  (Use: "git log
> > >>>>>>> flink-connectors/flink-connector-base")
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> commit
> > >>> 401f56fe9d6b0271260edf9787cdcbfe4d03874d
> > >>>>>>>>>>>>>>>>>>> [FLINK-19427][FLINK-19489][tests] Fix test
> > >>>>>> conditions
> > >>>>>>>>> for
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>> 'SplitFetcherTest.testNotifiesWhenGoingIdleConcurrent()'
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> commit
> > >>> 68c5c2ff779d82a1ff81ffaf60d8a1b283797db1
> > >>>>>>>>>>>>>>>>>>> [FLINK-19448][connector base] Explicitly
> > >> check
> > >>>> for
> > >>>>>>>>>>> un-expected
> > >>>>>>>>>>>>>>>>> condition
> > >>>>>>>>>>>>>>>>>>> that would leave an inconsistent state
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> commit
> > >>> 162c072e9265a7b6dd9d6f5459eb7974091c4c4e
> > >>>>>>>>>>>>>>>>>>> [FLINK-19492][core] Consolidate Source
> > >> Events
> > >>>>>> between
> > >>>>>>>>>> Source
> > >>>>>>>>>>>> API
> > >>>>>>>>>>>>>>> and
> > >>>>>>>>>>>>>>>>>> Split
> > >>>>>>>>>>>>>>>>>>> Reader API
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> commit
> > >>> c1ca7a4c7c21ec8868c14cdddd43c559625b794c
> > >>>>>>>>>>>>>>>>>>> [refactor][tests] Move some source test
> > >> utils
> > >>>> from
> > >>>>>>>>>>>>>>>>> flink-connector-base
> > >>>>>>>>>>>>>>>>>> to
> > >>>>>>>>>>>>>>>>>>> flink-core
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> commit
> > >>> ee5c4c211c35c70d28252363bbc8400453609977
> > >>>>>>>>>>>>>>>>>>> [FLINK-19251][connectors] Avoid confusing
> > >>> queue
> > >>>>>>>>> handling in
> > >>>>>>>>>>>>>>>>>>> "SplitReader.handleSplitsChanges()"
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> commit
> > >>> 5abef56b2bf85bcac786f6b16b6899b6cced7176
> > >>>>>>>>>>>>>>>>>>> [FLINK-19250][connectors] Fix error
> > >>> propagation
> > >>>> in
> > >>>>>>>>>> connector
> > >>>>>>>>>>>> base
> > >>>>>>>>>>>>>>>>>>> (SplitFetcherManager).
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> commit
> > >>> 8fcca837c55a9216595ee4c03038b52747098dbb
> > >>>>>>>>>>>>>>>>>>> [hotfix][connectors] Improve JavaDocs for
> > >>>>>>>>>>>>>>> SingleThreadFetcherManager
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> commit
> > >>> 4700bb5dde3303cbe98882f6beb7379425717b01
> > >>>>>>>>>>>>>>>>>>> [FLINK-19225][connectors] Various small
> > >>>>> improvements
> > >>>>>>> to
> > >>>>>>>>>>>>>>>>> SourceReaderBase
> > >>>>>>>>>>>>>>>>>>> (part 2)
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> commit
> > >>> 12261c6b7ed6478a9b9f6a69cb58246b83cab9b7
> > >>>>>>>>>>>>>>>>>>> [FLINK-17393][connectors] (follow-up) Wakeup
> > >>> the
> > >>>>>>>>>>> SplitFetchers
> > >>>>>>>>>>>>> more
> > >>>>>>>>>>>>>>>>>>> elegantly.
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> commit
> > >>> c60aaff0249bfd6b5871b7f82e03efc487a54d6b
> > >>>>>>>>>>>>>>>>>>> [hotfix][tests] Extend test coverage for
> > >>>>>>>>>>>>>>> FutureCompletingBlockingQueue
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> commit
> > >>> cef8a587d7fd2fe64cc644da5ed095d82e46f631
> > >>>>>>>>>>>>>>>>>>> [FLINK-19245][connectors] Set default
> > >> capacity
> > >>>> for
> > >>>>>>>>>>>>>>>>>>> FutureCompletingBlockingQueue.
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> commit
> > >>> 4ea95782b4c6a2538153d4d16ad3f4839c7de0fb
> > >>>>>>>>>>>>>>>>>>> [FLINK-19223][connectors] Simplify
> > >>> Availability
> > >>>>>> Future
> > >>>>>>>>>> Model
> > >>>>>>>>>>> in
> > >>>>>>>>>>>>>>> Base
> > >>>>>>>>>>>>>>>>>>> Connector
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> commit
> > >>> 511857049ba30c8ff0ee56da551fa4a479dc583e
> > >>>>>>>>>>>>>>>>>>> [FLINK-18128][connectors] Ensure idle split
> > >>>>> fetchers
> > >>>>>>>>> lead
> > >>>>>>>>>> to
> > >>>>>>>>>>>>>>>>> availability
> > >>>>>>>>>>>>>>>>>>> notifications.
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> commit
> > >>> a8206467af0830dcb89623ea068b5ca3b3450c92
> > >>>>>>>>>>>>>>>>>>> [refactor][core] Eagerly initialize the
> > >>>> FetchTask
> > >>>>> to
> > >>>>>>>>>> support
> > >>>>>>>>>>>>> proper
> > >>>>>>>>>>>>>>>>> unit
> > >>>>>>>>>>>>>>>>>>> testing
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> commit
> > >>> 3b2f54bcb437f98e6137c904045cc51072b5c06b
> > >>>>>>>>>>>>>>>>>>> [hotfix][tests] Move constants in
> > >>>> SplitFetcherTest
> > >>>>>>>>> relevant
> > >>>>>>>>>>> to
> > >>>>>>>>>>>>> only
> > >>>>>>>>>>>>>>>>> one
> > >>>>>>>>>>>>>>>>>>> test into test method
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> commit
> > >>> d7625760a75a508bf05bcddc380bb4d62ee1743e
> > >>>>>>>>>>>>>>>>>>> [FLINK-19225][connectors] Various small
> > >>>>> improvements
> > >>>>>>> to
> > >>>>>>>>>>>>>>>>> SourceReaderBase
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> commit
> > >>> a5b0d3297748c1be47ad579a88f24df2255a8df1
> > >>>>>>>>>>>>>>>>>>> [FLINK-17393][connectors] Wakeup the
> > >>>> SplitFetchers
> > >>>>>>> more
> > >>>>>>>>>>>>> elegantly.
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> commit
> > >>> f42a3ebc3e81a034b7221a803c153636fef34903
> > >>>>>>>>>>>>>>>>>>> [FLINK-18680][connectors] Make connector
> > >> base
> > >>>>>>>>>>>> RecordsWithSplitIds
> > >>>>>>>>>>>>>>> more
> > >>>>>>>>>>>>>>>>>>> lightweight.
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> commit
> > >>> e3d273de822b085183d09b275a445879ff94b350
> > >>>>>>>>>>>>>>>>>>> [FLINK-19162][connectors] Add 'recycle()' to
> > >>> the
> > >>>>>>>>>>>>>>> RecordsWithSplitIds
> > >>>>>>>>>>>>>>>>> to
> > >>>>>>>>>>>>>>>>>>> support reuse of heavy objects.
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> commit
> > >>> 8ebc464c2520453a70001cd712abc8dee6ee89e0
> > >>>>>>>>>>>>>>>>>>> [hotfix][testing] Add a set of
> > >> parameterizable
> > >>>>>> testing
> > >>>>>>>>>> mocks
> > >>>>>>>>>>>> for
> > >>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>> Split
> > >>>>>>>>>>>>>>>>>>> Reader API
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> commit
> > >>> 930a07438be1185388d7150640f294dfe2a5d378
> > >>>>>>>>>>>>>>>>>>> [hotfix][connectors] Add RequestSplitEvent
> > >> to
> > >>>>>>>>>>>>>>> 'flink-connector-base'
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> commit
> > >>> e945ce8a933bc378844782f784ca473c767ca159
> > >>>>>>>>>>>>>>>>>>> [refactor][connectors] Backport of the
> > >>>>>> connector-base
> > >>>>>>>>>>> exception
> > >>>>>>>>>>>>>>>>> handling
> > >>>>>>>>>>>>>>>>>>> from the Kafka Connector Pull Request
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> On Wed, Oct 28, 2020 at 5:49 PM Steven Wu <
> > >>>>>>>>>>>> stevenz3wu@gmail.com>
> > >>>>>>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>> I would love to see this FLIP-27 source
> > >>>>> interface
> > >>>>>>>>>>> improvement
> > >>>>>>>>>>>>> [1]
> > >>>>>>>>>>>>>>>>> made
> > >>>>>>>>>>>>>>>>>> to
> > >>>>>>>>>>>>>>>>>>>> 1.11.3.
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>> [1]
> > >>>>>>> https://issues.apache.org/jira/browse/FLINK-19698
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>> On Wed, Oct 28, 2020 at 12:32 AM Tzu-Li
> > >>>> (Gordon)
> > >>>>>>> Tai <
> > >>>>>>>>>>>>>>>>>>> tzulitai@apache.org>
> > >>>>>>>>>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>> Thanks for the replies so far!
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>> Just to provide a brief update on the
> > >>> status
> > >>>>> of
> > >>>>>>>>>> blockers
> > >>>>>>>>>>>> for
> > >>>>>>>>>>>>>>>>> 1.11.3
> > >>>>>>>>>>>>>>>>>> so
> > >>>>>>>>>>>>>>>>>>>> far:
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>> *PR opened, pending reviewer*-
> > >>> [FLINK-19717]
> > >>>>>>>>>>>>>>>>>> SourceReaderBase.pollNext
> > >>>>>>>>>>>>>>>>>>>> may
> > >>>>>>>>>>>>>>>>>>>>> return END_OF_INPUT if SplitReader.fetch
> > >>>>> throws
> > >>>>>> (
> > >>>>>>>>>>>>>>>>>>>>>
> > >>> https://github.com/apache/flink/pull/13776)
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>> *PR opened, reviewed + close to being
> > >>>> merged*
> > >>>>>>>>>>>>>>>>>>>>> - [FLINK-19741] Timer service should
> > >> skip
> > >>>>>>> restoring
> > >>>>>>>>>> from
> > >>>>>>>>>>>> raw
> > >>>>>>>>>>>>>>> keyed
> > >>>>>>>>>>>>>>>>>>> stream
> > >>>>>>>>>>>>>>>>>>>>> if it isn't the writer (
> > >>>>>>>>>>>>>>>>> https://github.com/apache/flink/pull/13761)
> > >>>>>>>>>>>>>>>>>>>>> - [FLINK-19748] Raw keyed stream key
> > >> group
> > >>>>>>> iterator
> > >>>>>>>>>>> should
> > >>>>>>>>>>>> be
> > >>>>>>>>>>>>>>>>>> skipping
> > >>>>>>>>>>>>>>>>>>>>> unwritten key groups (
> > >>>>>>>>>>>>>>> https://github.com/apache/flink/pull/13772)
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>> *Merged*
> > >>>>>>>>>>>>>>>>>>>>> - [FLINK-19154] Application mode deletes
> > >>> HA
> > >>>>> data
> > >>>>>>> in
> > >>>>>>>>>> case
> > >>>>>>>>>>> of
> > >>>>>>>>>>>>>>>>> suspended
> > >>>>>>>>>>>>>>>>>>>>> ZooKeeper connection
> > >>>>>>>>>>>>>>>>>>>>> - [FLINK-19569] Upgrade ICU4J to 67.1+
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>> Right now as it seems, progress is
> > >> mainly
> > >>>>>> blocked
> > >>>>>>>>> on a
> > >>>>>>>>>>>>> reviewer
> > >>>>>>>>>>>>>>>>> for
> > >>>>>>>>>>>>>>>>>>>>> FLINK-19717.
> > >>>>>>>>>>>>>>>>>>>>> Meanwhile, Xintong is keeping an eye on
> > >>> test
> > >>>>>>>>>>> instabilities
> > >>>>>>>>>>>>> [1]
> > >>>>>>>>>>>>>>> to
> > >>>>>>>>>>>>>>>>> see
> > >>>>>>>>>>>>>>>>>>> if
> > >>>>>>>>>>>>>>>>>>>>> there are any fixes that should be
> > >> applied
> > >>>> to
> > >>>>>>>>>>>> `release-1.11`.
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>> This is also a reminder, that if there
> > >> are
> > >>>>> other
> > >>>>>>>>>> blockers
> > >>>>>>>>>>>>> that
> > >>>>>>>>>>>>>>> we
> > >>>>>>>>>>>>>>>>>> need
> > >>>>>>>>>>>>>>>>>>> to
> > >>>>>>>>>>>>>>>>>>>>> be aware of, or a need to re-establish
> > >>>>> estimated
> > >>>>>>>>> time
> > >>>>>>>>>> for
> > >>>>>>>>>>>>>>> getting
> > >>>>>>>>>>>>>>>>>> fixes
> > >>>>>>>>>>>>>>>>>>>> in
> > >>>>>>>>>>>>>>>>>>>>> and delay the RC for 1.11.3, please do
> > >> let
> > >>>> us
> > >>>>>>> know!
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>> Cheers,
> > >>>>>>>>>>>>>>>>>>>>> Gordon
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>> [1]
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>
> > >>>>>>
> > >>>>>
> > >>>>
> > >>>
> > >>
> >
> https://issues.apache.org/jira/issues/?jql=project%20%3D%20FLINK%20AND%20status%20in%20(Open%2C%20%22In%20Progress%22%2C%20Reopened)%20AND%20affectedVersion%20in%20(1.11.0%2C%201.11.1%2C%201.11.2)%20AND%20labels%20%3D%20test-stability
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>> On Mon, Oct 26, 2020 at 9:43 PM Kostas
> > >>>>> Kloudas <
> > >>>>>>>>>>>>>>>>> kkloudas@apache.org>
> > >>>>>>>>>>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>> +1 for releasing Flink 1.11.3 as it
> > >>>>> contains a
> > >>>>>>>>> number
> > >>>>>>>>>>> of
> > >>>>>>>>>>>>>>>>> important
> > >>>>>>>>>>>>>>>>>>>>>> fixes and thanks Gordon and Xintong
> > >> for
> > >>>>>>>>> volunteering.
> > >>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>> Cheers,
> > >>>>>>>>>>>>>>>>>>>>>> Kostas
> > >>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>> On Mon, Oct 26, 2020 at 4:37 AM Yu Li
> > >> <
> > >>>>>>>>>>> carp84@gmail.com>
> > >>>>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>> +1 for releasing Flink 1.11.3, and
> > >>>> thanks
> > >>>>>>> Gordon
> > >>>>>>>>>> and
> > >>>>>>>>>>>>>>> Xintong
> > >>>>>>>>>>>>>>>>> for
> > >>>>>>>>>>>>>>>>>>>>>>> volunteering as our release
> > >> managers.
> > >>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>> Best Regards,
> > >>>>>>>>>>>>>>>>>>>>>>> Yu
> > >>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>> On Mon, 26 Oct 2020 at 09:45,
> > >> Xintong
> > >>>>> Song <
> > >>>>>>>>>>>>>>>>>> tonysong820@gmail.com>
> > >>>>>>>>>>>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>> Thanks Gordan for starting this
> > >>>>>> discussion.
> > >>>>>>>>>>>>>>>>>>>>>>>> My pleasure to help with the
> > >> release
> > >>>>>>> process.
> > >>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>> Thank you~
> > >>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>> Xintong Song
> > >>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>> On Fri, Oct 23, 2020 at 11:29 PM
> > >>> Till
> > >>>>>>>>> Rohrmann <
> > >>>>>>>>>>>>>>>>>>>> trohrmann@apache.org
> > >>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>> Thanks for starting this
> > >>> discussion
> > >>>>>>> Gordon.
> > >>>>>>>>>> There
> > >>>>>>>>>>>> are
> > >>>>>>>>>>>>>>> over
> > >>>>>>>>>>>>>>>>>> 100
> > >>>>>>>>>>>>>>>>>>>>> issues
> > >>>>>>>>>>>>>>>>>>>>>>>>> which are fixed for 1.11.3.
> > >> Hence
> > >>> +1
> > >>>>>> for a
> > >>>>>>>>>>> soonish
> > >>>>>>>>>>>>>>> 1.11.3
> > >>>>>>>>>>>>>>>>>>>> release.
> > >>>>>>>>>>>>>>>>>>>>>> Thanks
> > >>>>>>>>>>>>>>>>>>>>>>>>> for volunteering as our release
> > >>>>> managers
> > >>>>>>>>> Gordon
> > >>>>>>>>>>> and
> > >>>>>>>>>>>>>>>>> Xintong!
> > >>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>> Cheers,
> > >>>>>>>>>>>>>>>>>>>>>>>>> Till
> > >>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>> On Fri, Oct 23, 2020 at 5:02 PM
> > >>>> Tzu-Li
> > >>>>>>>>> (Gordon)
> > >>>>>>>>>>>> Tai <
> > >>>>>>>>>>>>>>>>>>>>>> tzulitai@apache.org
> > >>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>> Hi,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>> Xintong and I would like to
> > >>> start a
> > >>>>>>>>> discussion
> > >>>>>>>>>>> for
> > >>>>>>>>>>>>>>>>> releasing
> > >>>>>>>>>>>>>>>>>>>> Flink
> > >>>>>>>>>>>>>>>>>>>>>>>> 1.11.3
> > >>>>>>>>>>>>>>>>>>>>>>>>>> soon.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>> It seems like we already have a
> > >>> few
> > >>>>>>>>> pressing
> > >>>>>>>>>>>> issues
> > >>>>>>>>>>>>>>> that
> > >>>>>>>>>>>>>>>>>> needs
> > >>>>>>>>>>>>>>>>>>>> to
> > >>>>>>>>>>>>>>>>>>>>> be
> > >>>>>>>>>>>>>>>>>>>>>>>>>> included in a new hotfix
> > >> release:
> > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>   - Heap-based timers’ restore
> > >>>>>> behaviour
> > >>>>>>>>> is
> > >>>>>>>>>>>>> causing a
> > >>>>>>>>>>>>>>>>>>> critical
> > >>>>>>>>>>>>>>>>>>>>>> recovery
> > >>>>>>>>>>>>>>>>>>>>>>>>>>   issue for StateFun [1] [2]
> > >>> [3].
> > >>>>>>>>>>>>>>>>>>>>>>>>>>   - There are several
> > >> robustness
> > >>>>>> issues
> > >>>>>>>>> for
> > >>>>>>>>>> the
> > >>>>>>>>>>>>>>> FLIP-27
> > >>>>>>>>>>>>>>>>> new
> > >>>>>>>>>>>>>>>>>>>>> source
> > >>>>>>>>>>>>>>>>>>>>>> API,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>   such as [4]. We already have
> > >>>> some
> > >>>>>>> users
> > >>>>>>>>>> using
> > >>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>> FLIP-27
> > >>>>>>>>>>>>>>>>>>> API
> > >>>>>>>>>>>>>>>>>>>>>> with
> > >>>>>>>>>>>>>>>>>>>>>>>>>> 1.11.x,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>   so it would be important to
> > >>> get
> > >>>>>> those
> > >>>>>>>>> fixes
> > >>>>>>>>>>> in
> > >>>>>>>>>>>>> for
> > >>>>>>>>>>>>>>>>> 1.11.x
> > >>>>>>>>>>>>>>>>>>> as
> > >>>>>>>>>>>>>>>>>>>>>> well.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>> Apart from the issues that are
> > >>>>> already
> > >>>>>>>>> marked
> > >>>>>>>>>> as
> > >>>>>>>>>>>>>>> blocker
> > >>>>>>>>>>>>>>>>> for
> > >>>>>>>>>>>>>>>>>>>>> 1.11.3
> > >>>>>>>>>>>>>>>>>>>>>> in
> > >>>>>>>>>>>>>>>>>>>>>>>> our
> > >>>>>>>>>>>>>>>>>>>>>>>>>> JIRA [5], please let us know in
> > >>>> this
> > >>>>>>>>> thread if
> > >>>>>>>>>>>> there
> > >>>>>>>>>>>>>>> is
> > >>>>>>>>>>>>>>>>>>> already
> > >>>>>>>>>>>>>>>>>>>>>> ongoing
> > >>>>>>>>>>>>>>>>>>>>>>>>>> work for other important fixes
> > >>> that
> > >>>>> we
> > >>>>>>>>> should
> > >>>>>>>>>>> try
> > >>>>>>>>>>>> to
> > >>>>>>>>>>>>>>>>>> include.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>> Xintong and I would like to
> > >>>> volunteer
> > >>>>>> for
> > >>>>>>>>>>> managing
> > >>>>>>>>>>>>>>> this
> > >>>>>>>>>>>>>>>>>>> release,
> > >>>>>>>>>>>>>>>>>>>>> and
> > >>>>>>>>>>>>>>>>>>>>>>>> will
> > >>>>>>>>>>>>>>>>>>>>>>>>>> try to communicate the priority
> > >>> of
> > >>>>>>> pending
> > >>>>>>>>>>>> blockers
> > >>>>>>>>>>>>>>> over
> > >>>>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>>> next
> > >>>>>>>>>>>>>>>>>>>>>> few
> > >>>>>>>>>>>>>>>>>>>>>>>>>> days.
> > >>>>>>>>>>>>>>>>>>>>>>>>>> Since the aforementioned issues
> > >>> are
> > >>>>>> quite
> > >>>>>>>>>>>> critical,
> > >>>>>>>>>>>>>>> we’d
> > >>>>>>>>>>>>>>>>>> like
> > >>>>>>>>>>>>>>>>>>> to
> > >>>>>>>>>>>>>>>>>>>>> aim
> > >>>>>>>>>>>>>>>>>>>>>>>>>> for a *feature
> > >>>>>>>>>>>>>>>>>>>>>>>>>> freeze by the end of next week
> > >>>> (Oct.
> > >>>>>>> 30th)*
> > >>>>>>>>>> and
> > >>>>>>>>>>>>> start
> > >>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>>> release
> > >>>>>>>>>>>>>>>>>>>>>> voting
> > >>>>>>>>>>>>>>>>>>>>>>>>>> process the week after.
> > >>>>>>>>>>>>>>>>>>>>>>>>>> If that is too short of a
> > >> notice
> > >>>> and
> > >>>>>> you
> > >>>>>>>>> might
> > >>>>>>>>>>>> need
> > >>>>>>>>>>>>>>> more
> > >>>>>>>>>>>>>>>>>> time,
> > >>>>>>>>>>>>>>>>>>>>>> please
> > >>>>>>>>>>>>>>>>>>>>>>>> let
> > >>>>>>>>>>>>>>>>>>>>>>>>>> us know!
> > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>> Cheers,
> > >>>>>>>>>>>>>>>>>>>>>>>>>> Gordon
> > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>> [1]
> > >>>>>>>>>>>>> https://issues.apache.org/jira/browse/FLINK-19692
> > >>>>>>>>>>>>>>>>>>>>>>>>>> [2]
> > >>>>>>>>>>>>> https://issues.apache.org/jira/browse/FLINK-19741
> > >>>>>>>>>>>>>>>>>>>>>>>>>> [3]
> > >>>>>>>>>>>>> https://issues.apache.org/jira/browse/FLINK-19748
> > >>>>>>>>>>>>>>>>>>>>>>>>>> [4]
> > >>>>>>>>>>>>> https://issues.apache.org/jira/browse/FLINK-19717
> > >>>>>>>>>>>>>>>>>>>>>>>>>> [5]
> > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>
> > >>>>>>
> > >>>>>
> > >>>>
> > >>>
> > >>
> >
> https://issues.apache.org/jira/issues/?jql=project%20%3D%20FLINK%20AND%20priority%20%3D%20Blocker%20AND%20fixVersion%20%3D%201.11.3
> > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> --
> > >>>>>>>>>>>>>>>>>> Best, Jingsong Lee
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>
> > >>>>>>>
> > >>>>>>
> > >>>>>
> > >>>>
> > >>>
> > >>
> >
> >
> >
>

Re: [DISCUSS] Releasing Apache Flink 1.11.3

Posted by Xintong Song <to...@gmail.com>.
Thanks for the notice, Dian.

Thank you~

Xintong Song



On Tue, Nov 10, 2020 at 10:19 AM Dian Fu <di...@gmail.com> wrote:

> Hi Xintong,
>
> I want to bring one more issue to your attention [1]. The test case
> UnalignedCheckpointCompatibilityITCase.test failed several times in the
> last nightly test of release-1.11. We need to figure out if it's just an
> instable test or caused by recent changes.
>
> [1] https://issues.apache.org/jira/browse/FLINK-20065
>
> > 在 2020年11月10日，上午9:24，Xintong Song <to...@gmail.com> 写道：
> >
> > Thanks for the replies.
> >
> > Thank you~
> >
> > Xintong Song
> >
> >
> >
> > On Tue, Nov 10, 2020 at 1:09 AM Becket Qin <be...@gmail.com> wrote:
> >
> >> Hi Xintong,
> >>
> >> Thanks for driving the release. Just want to sync up on the FLIP-27
> >> backporting. Stephan and I are still trying to backport a bunch of
> patches
> >> of Source to 1.11.3. Including:
> >>
> >> [FLINK-19698][connector/common] Add a close() method to the SplitReader.
> >> [FLINK-19717] SourceReaderBase.pollNext may return END_OF_INPUT if
> >> SplitReader.fetch throws
> >> [FLINK-19535] [connector/common] Avoid failing a job multiple times in
> >> SourceCoordinator.
> >> [FLINK-19265] [FLINK-20049][core] Source API final adjustments.
> >>
> >> and a few more fixes.
> >>
> >> We are currently trying to fix them in 1.12 first so it might take a
> little
> >> longer to backport them to 1.11.3. I think it will probably take us a
> few
> >> more days to finish the backport. So that would roughly be the end of
> this
> >> week.
> >>
> >> Thanks,
> >>
> >> Jiangjie (Becket) Qin
> >>
> >>
> >>
> >>
> >> On Mon, Nov 9, 2020 at 9:57 PM Till Rohrmann <tr...@apache.org>
> wrote:
> >>
> >>> Yes, I've downgraded FLINK-19816 to critical.
> >>>
> >>> Cheers,
> >>> Till
> >>>
> >>> On Mon, Nov 9, 2020 at 10:19 AM Xintong Song <to...@gmail.com>
> >>> wrote:
> >>>
> >>>> Thanks for the notice, Till.
> >>>>
> >>>> I just checked and found FLINK-20033 is already fixed. Shall we also
> >>>> downgrade FLINK-19816 to `Critical`?
> >>>>
> >>>> Thank you~
> >>>>
> >>>> Xintong Song
> >>>>
> >>>>
> >>>>
> >>>> On Mon, Nov 9, 2020 at 4:42 PM Till Rohrmann <tr...@apache.org>
> >>> wrote:
> >>>>
> >>>>> I would like to bring one more critical issue to your attention which
> >>> is
> >>>>> FLINK-20033 [1]. I believe that this issue is actually causing what
> >> has
> >>>>> been reported in FLINK-19816 [2]. I hope to have it fixed by the end
> >> of
> >>>>> today. Once FLINK-20033 is fixed, I think that we don't have to block
> >>> the
> >>>>> release on FLINK-19816.
> >>>>>
> >>>>> [1] https://issues.apache.org/jira/browse/FLINK-20033
> >>>>> [2] https://issues.apache.org/jira/browse/FLINK-19816
> >>>>>
> >>>>> Cheers,
> >>>>> Till
> >>>>>
> >>>>> On Mon, Nov 9, 2020 at 4:05 AM Xintong Song <to...@gmail.com>
> >>>> wrote:
> >>>>>
> >>>>>> Hi devs,
> >>>>>>
> >>>>>> I'd like to provide an update on the progress of preparing release
> >>>>> 1.11.3.
> >>>>>>
> >>>>>> *Blockers*
> >>>>>> We currently have 3 remaining blockers. (3 resolved and 1 emerged
> >>>>> compared
> >>>>>> to last week)
> >>>>>>
> >>>>>> - [FLINK-19698] Add close() method and onCheckpointComplete() to
> >> the
> >>>>>> Source.
> >>>>>> The issue has been fixed on the master branch. It's currently
> >> blocked
> >>>> on
> >>>>>> the FLIP-27 backportings to backport it to the 1.11 branch.
> >>>>>>
> >>>>>> - [FLINK-19717] SourceReaderBase.pollNext may return END_OF_INPUT
> >> if
> >>>>>> SplitReader.fetch throws
> >>>>>> A PR has been opened and reviewed. From the discussions on the PR,
> >> it
> >>>>> looks
> >>>>>> close to mergeable.
> >>>>>>
> >>>>>> - [FLINK-19816] Flink restored from a wrong checkpoint (a very old
> >>> one
> >>>>> and
> >>>>>> not the last completed one)
> >>>>>> This is a newly emerged blocker and Matthias is working on it.
> >>>>>>
> >>>>>> *Test Instabilities*
> >>>>>> We currently have 27 test instabilities[1].
> >>>>>> AFAIK, none of them are as serious as to block the 1.11.3 release.
> >>>>>>
> >>>>>> *FLIP-27 Backprotings*
> >>>>>>
> >>>>>> I noticed that there's no jira issues opened on the FLIP-27
> >>> backporting
> >>>>>> efforts, which is part of the major efforts planned for the 1.11.3
> >>>>> release,
> >>>>>> making it hard to track the progress.
> >>>>>>
> >>>>>>
> >>>>>> @Stephan and @Becket, could you please share the updates on the
> >>>>> backporting
> >>>>>> efforts? How is the progress and when are the efforts expected to
> >> be
> >>>>>> finished? It would be appreciated and helpful if we can have a jira
> >>>>> ticket
> >>>>>> to track the backporing progress publicly.
> >>>>>>
> >>>>>>
> >>>>>> Thank you~
> >>>>>>
> >>>>>> Xintong Song
> >>>>>>
> >>>>>>
> >>>>>> [1]
> >>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>>
> >>
> https://issues.apache.org/jira/browse/FLINK-16908?filter=12348580&jql=project%20%3D%20FLINK%20AND%20resolution%20%3D%20Unresolved%20AND%20affectedVersion%20in%20(1.11.0%2C%201.11.1%2C%201.11.2%2C%201.11.3)%20AND%20labels%20%3D%20test-stability%20ORDER%20BY%20updated%20DESC%2C%20created%20DESC
> >>>>>>
> >>>>>> On Wed, Nov 4, 2020 at 3:58 PM Tzu-Li (Gordon) Tai <
> >>>> tzulitai@apache.org>
> >>>>>> wrote:
> >>>>>>
> >>>>>>> Thanks for the efforts so far with pushing for 1.11.3.
> >>>>>>>
> >>>>>>> I'd like to provide a mid-week update on how we're looking with
> >> all
> >>>> the
> >>>>>>> pending blockers and backports:
> >>>>>>>
> >>>>>>> *Blockers**:*
> >>>>>>>
> >>>>>>> Currently all blockers either have an open PR, or have been
> >> merged.
> >>>>> I'll
> >>>>>>> highlight below the blockers *that still require reviewing
> >> efforts
> >>>> for
> >>>>>> them
> >>>>>>> to move forward:*
> >>>>>>>
> >>>>>>> - [FLINK-19909] Flink application in attach mode could not
> >>> terminate
> >>>>> when
> >>>>>>> the only job is canceled. PR:
> >>>>> https://github.com/apache/flink/pull/13911
> >>>>>>> - [FLINK-19717] SourceReaderBase.pollNext may return END_OF_INPUT
> >>> if
> >>>>>>> SplitReader.fetch throws. PR:
> >>>>> https://github.com/apache/flink/pull/13776
> >>>>>>>
> >>>>>>> The above PRs currently have no reviews at all yet. They seem to
> >>>>> already
> >>>>>>> have designated reviewers.
> >>>>>>>
> >>>>>>>
> >>>>>>> *Backports:*
> >>>>>>> - FLIP-27 robustness improvement backports: Stephan is currently
> >>>>> working
> >>>>>> on
> >>>>>>> backporting several FLIP-27 changes. There are no PRs yet for the
> >>>>>>> backported changes.
> >>>>>>> - DataStreamUtils.collect() refactorings backport: Steven Wu
> >>>> mentioned
> >>>>> to
> >>>>>>> backport this, but AFAIK this isn't assigned to anyone yet.
> >>>>>>>
> >>>>>>> *ETAs*:
> >>>>>>>
> >>>>>>> I'd like to request ETAs for the remaining backports, to prevent
> >> a
> >>>>> creep
> >>>>>> in
> >>>>>>> the scope of this bugfix release.
> >>>>>>>
> >>>>>>> We already have Flink users that would benefit from fixes that
> >> have
> >>>>> been
> >>>>>>> merged to release-1.11 already, so technically speaking the
> >>> backports
> >>>>>>> should be considered "nice-to-have" (to the best of my knowledge
> >> of
> >>>> the
> >>>>>>> changes) and could potentially be moved to a follow-up 1.11.4.
> >>>>>>> Most notably, the Stateful Functions project is already waiting
> >> on
> >>>>> Flink
> >>>>>>> 1.11.3 to address critical recovery issues (please see the
> >> StateFun
> >>>>> 2.2.1
> >>>>>>> release discussion thread [1]).
> >>>>>>>
> >>>>>>> @Stephan Ewen <se...@apache.org> @Becket Qin <
> >> becket.qin@gmail.com
> >>>>
> >>>>>> could
> >>>>>>> you provide an ETA for the FLIP-27 backports? It would help to
> >> get
> >>> a
> >>>>>> better
> >>>>>>> estimate to decide how we proceed here.
> >>>>>>>
> >>>>>>> Cheers,
> >>>>>>> Gordon
> >>>>>>>
> >>>>>>> [1]
> >>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>>
> >>
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Releasing-StateFun-hotfix-version-2-2-1-td46239.html
> >>>>>>>
> >>>>>>> On Wed, Nov 4, 2020 at 3:16 PM Tzu-Li (Gordon) Tai <
> >>>>> tzulitai@apache.org>
> >>>>>>> wrote:
> >>>>>>>
> >>>>>>>>> The collect() utils can be picked back, I see no issue with
> >>> that
> >>>>> (it
> >>>>>> is
> >>>>>>>> isolated utilities).
> >>>>>>>>
> >>>>>>>> Just checking on all the requested backports mentioned in this
> >>>>> thread,
> >>>>>>> and
> >>>>>>>> figuring out which ones seem to still be unassigned / open.
> >>>>>>>>
> >>>>>>>> Is someone working on backporting
> >>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>>
> >>
> https://github.com/apache/flink/commit/09a7a66b7313fea64817fe960a8da1265b428efc
> >>>>>>>> to release-1.11 at the moment?
> >>>>>>>>
> >>>>>>>> On Wed, Nov 4, 2020 at 1:39 AM Steven Wu <stevenz3wu@gmail.com
> >>>
> >>>>> wrote:
> >>>>>>>>
> >>>>>>>>> @Stephan Ewen <se...@apache.org> yeah, we can do that. don't
> >>>> worry
> >>>>>>> about
> >>>>>>>>> it. your earlier email had the perfect explanation on why file
> >>>>> source
> >>>>>>>>> shouldn't be backported.
> >>>>>>>>>
> >>>>>>>>> On Tue, Nov 3, 2020 at 3:37 AM Stephan Ewen <sewen@apache.org
> >>>
> >>>>> wrote:
> >>>>>>>>>
> >>>>>>>>>> @Steven would it be possible to initially copy some of the
> >>> code
> >>>>> into
> >>>>>>> the
> >>>>>>>>>> iceberg source and later replace it by a dependency on the
> >>> Flink
> >>>>>> file
> >>>>>>>>>> source?
> >>>>>>>>>>
> >>>>>>>>>> On Mon, Nov 2, 2020 at 8:33 PM Steven Wu <
> >>> stevenz3wu@gmail.com>
> >>>>>>> wrote:
> >>>>>>>>>>
> >>>>>>>>>>> Stephan, thanks a lot for explaining the file connector.
> >>> that
> >>>>>> makes
> >>>>>>>>>> sense.
> >>>>>>>>>>>
> >>>>>>>>>>> I was asking because we were trying to reuse some of the
> >>>>>>>>> implementations
> >>>>>>>>>> in
> >>>>>>>>>>> the file source for Iceberg source. Flink Iceberg source
> >>> lives
> >>>>> in
> >>>>>>> the
> >>>>>>>>>>> Iceberg repo, which is not possible to code against the
> >>> master
> >>>>>>> branch
> >>>>>>>>> of
> >>>>>>>>>>> the Flink code.
> >>>>>>>>>>>
> >>>>>>>>>>> On Mon, Nov 2, 2020 at 3:31 AM Stephan Ewen <
> >>> sewen@apache.org
> >>>>>
> >>>>>>> wrote:
> >>>>>>>>>>>
> >>>>>>>>>>>> Hi Steven!
> >>>>>>>>>>>>
> >>>>>>>>>>>> So far there are no plans to pick back the file system
> >>>>> connector
> >>>>>>>>> code.
> >>>>>>>>>>> This
> >>>>>>>>>>>> is still evolving and not finalized for 1.12, so I don't
> >>>> feel
> >>>>> it
> >>>>>>> is
> >>>>>>>>> a
> >>>>>>>>>>> good
> >>>>>>>>>>>> candidate to be backported.
> >>>>>>>>>>>> However, with the base connector changes backported, you
> >>>>> should
> >>>>>> be
> >>>>>>>>> able
> >>>>>>>>>>> to
> >>>>>>>>>>>> run the file connector code from master against 1.11.3.
> >>>>>>>>>>>>
> >>>>>>>>>>>> The collect() utils can be picked back, I see no issue
> >>> with
> >>>>> that
> >>>>>>>>> (it is
> >>>>>>>>>>>> isolated utilities).
> >>>>>>>>>>>>
> >>>>>>>>>>>> Best,
> >>>>>>>>>>>> Stephan
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> On Mon, Nov 2, 2020 at 3:02 AM Steven Wu <
> >>>>> stevenz3wu@gmail.com>
> >>>>>>>>> wrote:
> >>>>>>>>>>>>
> >>>>>>>>>>>>> Basically, it would be great to get the latest code in
> >>> the
> >>>>>>>>>>>>> flink-connector-files (FLIP-27).
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> On Sat, Oct 31, 2020 at 9:57 AM Steven Wu <
> >>>>>> stevenz3wu@gmail.com
> >>>>>>>>
> >>>>>>>>>>> wrote:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>> Stephan, it will be great if we can also backport
> >> the
> >>>>>>>>>> DataStreamUtils
> >>>>>>>>>>>>>> related commits that help with collecting output
> >> from
> >>>>>>> unbounded
> >>>>>>>>>>>> streams.
> >>>>>>>>>>>>>> e.g.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>>
> >>
> https://github.com/apache/flink/commit/09a7a66b7313fea64817fe960a8da1265b428efc
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> I tried to copy and paste the code to unblock
> >> myself.
> >>>> but
> >>>>> it
> >>>>>>>>>> quickly
> >>>>>>>>>>>> got
> >>>>>>>>>>>>>> into the rabbit hole of more and more code.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> On Fri, Oct 30, 2020 at 11:02 AM Stephan Ewen <
> >>>>>>> sewen@apache.org
> >>>>>>>>>>
> >>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> I have started with backporting the source API
> >>> changes.
> >>>>>> Some
> >>>>>>>>> minor
> >>>>>>>>>>>>>>> conflicts to solve, will need a bit more to finish
> >>>> this.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> On Fri, Oct 30, 2020 at 7:25 AM Tzu-Li (Gordon)
> >> Tai <
> >>>>>>>>>>>>> tzulitai@apache.org>
> >>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> @Stephan Ewen <se...@apache.org>
> >>>>>>>>>>>>>>>> Are there already plans or ongoing efforts for
> >>>>>> backporting
> >>>>>>>>> the
> >>>>>>>>>>> list
> >>>>>>>>>>>> of
> >>>>>>>>>>>>>>>> FLIP-27 changes that you posted?
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> On Thu, Oct 29, 2020 at 7:08 PM Xintong Song <
> >>>>>>>>>>> tonysong820@gmail.com
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Hi folks,
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Just to provide some updates concerning the
> >> status
> >>>> on
> >>>>>> the
> >>>>>>>>>>>>>>>>> test instabilities.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Currently, we have 30 unresolved tickets labeled
> >>>> with
> >>>>>>>>> `Affects
> >>>>>>>>>>>>> Version`
> >>>>>>>>>>>>>>>>> 1.11.x.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>>
> >>
> https://issues.apache.org/jira/browse/FLINK-19775?filter=12348580&jql=project%20%3D%20FLINK%20AND%20resolution%20%3D%20Unresolved%20AND%20affectedVersion%20in%20(1.11.0%2C%201.11.1%2C%201.11.2%2C%201.11.3)%20AND%20labels%20%3D%20test-stability%20ORDER%20BY%20created%20DESC
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Among the 30 tickets, 11 of them are:
> >>>>>>>>>>>>>>>>> - Have occured in the recent 3 months
> >>>>>>>>>>>>>>>>> - Not confirmed to be pure testability issues
> >>>>>>>>>>>>>>>>> - Not confirmed to be rare condition cases
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> It would be nice if someone familiar with these
> >>>>>> components
> >>>>>>>>> can
> >>>>>>>>>>>> take a
> >>>>>>>>>>>>>>> look
> >>>>>>>>>>>>>>>>> into these issues.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> -
> >>> https://issues.apache.org/jira/browse/FLINK-17159
> >>>>>> (ES6)
> >>>>>>>>>>>>>>>>> -
> >>> https://issues.apache.org/jira/browse/FLINK-17912
> >>>>>>> (Kafka)
> >>>>>>>>>>>>>>>>> -
> >>> https://issues.apache.org/jira/browse/FLINK-17949
> >>>>>>> (Kafka)
> >>>>>>>>>>>>>>>>> ⁃
> >>> https://issues.apache.org/jira/browse/FLINK-18444
> >>>>>>> (Kafka)
> >>>>>>>>>>>>>>>>> -
> >>> https://issues.apache.org/jira/browse/FLINK-18634
> >>>>>>> (Kafka)
> >>>>>>>>>>>>>>>>> -
> >>> https://issues.apache.org/jira/browse/FLINK-18648
> >>>>>>> (Kafka)
> >>>>>>>>>>>>>>>>> -
> >>> https://issues.apache.org/jira/browse/FLINK-18807
> >>>>>>> (Kafka)
> >>>>>>>>>>>>>>>>> -
> >>> https://issues.apache.org/jira/browse/FLINK-19369
> >>>>>>>>>>>> (BlobClientTest)
> >>>>>>>>>>>>>>>>> -
> >>> https://issues.apache.org/jira/browse/FLINK-19436
> >>>>>>> (TPCDS)
> >>>>>>>>>>>>>>>>> -
> >>> https://issues.apache.org/jira/browse/FLINK-19690
> >>>>>>>>>>>> (Format/Parquet)
> >>>>>>>>>>>>>>>>> -
> >>> https://issues.apache.org/jira/browse/FLINK-19775
> >>>>>>>>>>>>>>>>> (SystemProcessingTimeServiceTest)
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Thank you~
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Xintong Song
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> On Thu, Oct 29, 2020 at 10:21 AM Jingsong Li <
> >>>>>>>>>>>> jingsonglee0@gmail.com
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> +1 to backport the FLIP-27 adjustments to
> >>> 1.11.x.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> If possible, that would be great. Many people
> >>> are
> >>>>>>> looking
> >>>>>>>>>>> forward
> >>>>>>>>>>>>> to
> >>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>> FLIP-27 interface, but they don't want to take
> >>> the
> >>>>>> risk
> >>>>>>> to
> >>>>>>>>>>>> upgrade
> >>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>> 1.12
> >>>>>>>>>>>>>>>>>> (And wait 1.12). After all, 1.11 is a
> >> relatively
> >>>>>> stable
> >>>>>>>>>>> version.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Best,
> >>>>>>>>>>>>>>>>>> Jingsong
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> On Thu, Oct 29, 2020 at 1:24 AM Stephan Ewen <
> >>>>>>>>>> sewen@apache.org
> >>>>>>>>>>>>
> >>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Thanks for starting this.
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> +1 form my side to backport the FLIP-27
> >>>>> adjustments
> >>>>>> to
> >>>>>>>>>>> 1.11.x.
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> There were quite a few changes, and I think
> >> we
> >>>>> need
> >>>>>> to
> >>>>>>>>>>>>> cherry-pick
> >>>>>>>>>>>>>>>>> them
> >>>>>>>>>>>>>>>>>> all
> >>>>>>>>>>>>>>>>>>> to not get some inconsistent mix of changes
> >>> and
> >>>>> many
> >>>>>>>>> merge
> >>>>>>>>>>>>>>> conflicts.
> >>>>>>>>>>>>>>>>>>> I made a list below of what we need to add
> >> to
> >>>>>>>>>> "release-1.11".
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> *#### Core Source API Changes to backport
> >> (in
> >>>>>> REVERSE
> >>>>>>>>>> order)*
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>  (Use: "git log
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>> flink-core/src/main/java/org/apache/flink/api/connector/source")
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> commit
> >>> 162c072e9265a7b6dd9d6f5459eb7974091c4c4e
> >>>>>>>>>>>>>>>>>>> [FLINK-19492][core] Consolidate Source
> >> Events
> >>>>>> between
> >>>>>>>>>> Source
> >>>>>>>>>>>> API
> >>>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>>> Split
> >>>>>>>>>>>>>>>>>>> Reader API
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> commit
> >>> 428fbbad85379f71a9fbf1a1ab3e3dcfa9006436
> >>>>>>>>>>>>>>>>>>> [FLINK-19205][core] Add access to
> >>> configuration
> >>>>> and
> >>>>>>>>>> hostname
> >>>>>>>>>>> in
> >>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>> SourceReaderContext
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> commit
> >>> 723e1790744ddc78d19c7c978442af1383f38d33
> >>>>>>>>>>>>>>>>>>> [hotfix][core] Add to Source Enumerator
> >>>>> convenience
> >>>>>>>>> methods
> >>>>>>>>>>> to
> >>>>>>>>>>>>>>> assign
> >>>>>>>>>>>>>>>>>>> single split
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> *#### Connector Base Changes to Backport (in
> >>>>> REVERSE
> >>>>>>>>>> order)*
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>  (Use: "git log
> >>>>>>> flink-connectors/flink-connector-base")
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> commit
> >>> 401f56fe9d6b0271260edf9787cdcbfe4d03874d
> >>>>>>>>>>>>>>>>>>> [FLINK-19427][FLINK-19489][tests] Fix test
> >>>>>> conditions
> >>>>>>>>> for
> >>>>>>>>>>>>>>>>>>>
> >>>>>>> 'SplitFetcherTest.testNotifiesWhenGoingIdleConcurrent()'
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> commit
> >>> 68c5c2ff779d82a1ff81ffaf60d8a1b283797db1
> >>>>>>>>>>>>>>>>>>> [FLINK-19448][connector base] Explicitly
> >> check
> >>>> for
> >>>>>>>>>>> un-expected
> >>>>>>>>>>>>>>>>> condition
> >>>>>>>>>>>>>>>>>>> that would leave an inconsistent state
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> commit
> >>> 162c072e9265a7b6dd9d6f5459eb7974091c4c4e
> >>>>>>>>>>>>>>>>>>> [FLINK-19492][core] Consolidate Source
> >> Events
> >>>>>> between
> >>>>>>>>>> Source
> >>>>>>>>>>>> API
> >>>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>>> Split
> >>>>>>>>>>>>>>>>>>> Reader API
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> commit
> >>> c1ca7a4c7c21ec8868c14cdddd43c559625b794c
> >>>>>>>>>>>>>>>>>>> [refactor][tests] Move some source test
> >> utils
> >>>> from
> >>>>>>>>>>>>>>>>> flink-connector-base
> >>>>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>> flink-core
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> commit
> >>> ee5c4c211c35c70d28252363bbc8400453609977
> >>>>>>>>>>>>>>>>>>> [FLINK-19251][connectors] Avoid confusing
> >>> queue
> >>>>>>>>> handling in
> >>>>>>>>>>>>>>>>>>> "SplitReader.handleSplitsChanges()"
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> commit
> >>> 5abef56b2bf85bcac786f6b16b6899b6cced7176
> >>>>>>>>>>>>>>>>>>> [FLINK-19250][connectors] Fix error
> >>> propagation
> >>>> in
> >>>>>>>>>> connector
> >>>>>>>>>>>> base
> >>>>>>>>>>>>>>>>>>> (SplitFetcherManager).
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> commit
> >>> 8fcca837c55a9216595ee4c03038b52747098dbb
> >>>>>>>>>>>>>>>>>>> [hotfix][connectors] Improve JavaDocs for
> >>>>>>>>>>>>>>> SingleThreadFetcherManager
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> commit
> >>> 4700bb5dde3303cbe98882f6beb7379425717b01
> >>>>>>>>>>>>>>>>>>> [FLINK-19225][connectors] Various small
> >>>>> improvements
> >>>>>>> to
> >>>>>>>>>>>>>>>>> SourceReaderBase
> >>>>>>>>>>>>>>>>>>> (part 2)
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> commit
> >>> 12261c6b7ed6478a9b9f6a69cb58246b83cab9b7
> >>>>>>>>>>>>>>>>>>> [FLINK-17393][connectors] (follow-up) Wakeup
> >>> the
> >>>>>>>>>>> SplitFetchers
> >>>>>>>>>>>>> more
> >>>>>>>>>>>>>>>>>>> elegantly.
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> commit
> >>> c60aaff0249bfd6b5871b7f82e03efc487a54d6b
> >>>>>>>>>>>>>>>>>>> [hotfix][tests] Extend test coverage for
> >>>>>>>>>>>>>>> FutureCompletingBlockingQueue
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> commit
> >>> cef8a587d7fd2fe64cc644da5ed095d82e46f631
> >>>>>>>>>>>>>>>>>>> [FLINK-19245][connectors] Set default
> >> capacity
> >>>> for
> >>>>>>>>>>>>>>>>>>> FutureCompletingBlockingQueue.
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> commit
> >>> 4ea95782b4c6a2538153d4d16ad3f4839c7de0fb
> >>>>>>>>>>>>>>>>>>> [FLINK-19223][connectors] Simplify
> >>> Availability
> >>>>>> Future
> >>>>>>>>>> Model
> >>>>>>>>>>> in
> >>>>>>>>>>>>>>> Base
> >>>>>>>>>>>>>>>>>>> Connector
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> commit
> >>> 511857049ba30c8ff0ee56da551fa4a479dc583e
> >>>>>>>>>>>>>>>>>>> [FLINK-18128][connectors] Ensure idle split
> >>>>> fetchers
> >>>>>>>>> lead
> >>>>>>>>>> to
> >>>>>>>>>>>>>>>>> availability
> >>>>>>>>>>>>>>>>>>> notifications.
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> commit
> >>> a8206467af0830dcb89623ea068b5ca3b3450c92
> >>>>>>>>>>>>>>>>>>> [refactor][core] Eagerly initialize the
> >>>> FetchTask
> >>>>> to
> >>>>>>>>>> support
> >>>>>>>>>>>>> proper
> >>>>>>>>>>>>>>>>> unit
> >>>>>>>>>>>>>>>>>>> testing
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> commit
> >>> 3b2f54bcb437f98e6137c904045cc51072b5c06b
> >>>>>>>>>>>>>>>>>>> [hotfix][tests] Move constants in
> >>>> SplitFetcherTest
> >>>>>>>>> relevant
> >>>>>>>>>>> to
> >>>>>>>>>>>>> only
> >>>>>>>>>>>>>>>>> one
> >>>>>>>>>>>>>>>>>>> test into test method
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> commit
> >>> d7625760a75a508bf05bcddc380bb4d62ee1743e
> >>>>>>>>>>>>>>>>>>> [FLINK-19225][connectors] Various small
> >>>>> improvements
> >>>>>>> to
> >>>>>>>>>>>>>>>>> SourceReaderBase
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> commit
> >>> a5b0d3297748c1be47ad579a88f24df2255a8df1
> >>>>>>>>>>>>>>>>>>> [FLINK-17393][connectors] Wakeup the
> >>>> SplitFetchers
> >>>>>>> more
> >>>>>>>>>>>>> elegantly.
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> commit
> >>> f42a3ebc3e81a034b7221a803c153636fef34903
> >>>>>>>>>>>>>>>>>>> [FLINK-18680][connectors] Make connector
> >> base
> >>>>>>>>>>>> RecordsWithSplitIds
> >>>>>>>>>>>>>>> more
> >>>>>>>>>>>>>>>>>>> lightweight.
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> commit
> >>> e3d273de822b085183d09b275a445879ff94b350
> >>>>>>>>>>>>>>>>>>> [FLINK-19162][connectors] Add 'recycle()' to
> >>> the
> >>>>>>>>>>>>>>> RecordsWithSplitIds
> >>>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>> support reuse of heavy objects.
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> commit
> >>> 8ebc464c2520453a70001cd712abc8dee6ee89e0
> >>>>>>>>>>>>>>>>>>> [hotfix][testing] Add a set of
> >> parameterizable
> >>>>>> testing
> >>>>>>>>>> mocks
> >>>>>>>>>>>> for
> >>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>> Split
> >>>>>>>>>>>>>>>>>>> Reader API
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> commit
> >>> 930a07438be1185388d7150640f294dfe2a5d378
> >>>>>>>>>>>>>>>>>>> [hotfix][connectors] Add RequestSplitEvent
> >> to
> >>>>>>>>>>>>>>> 'flink-connector-base'
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> commit
> >>> e945ce8a933bc378844782f784ca473c767ca159
> >>>>>>>>>>>>>>>>>>> [refactor][connectors] Backport of the
> >>>>>> connector-base
> >>>>>>>>>>> exception
> >>>>>>>>>>>>>>>>> handling
> >>>>>>>>>>>>>>>>>>> from the Kafka Connector Pull Request
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> On Wed, Oct 28, 2020 at 5:49 PM Steven Wu <
> >>>>>>>>>>>> stevenz3wu@gmail.com>
> >>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> I would love to see this FLIP-27 source
> >>>>> interface
> >>>>>>>>>>> improvement
> >>>>>>>>>>>>> [1]
> >>>>>>>>>>>>>>>>> made
> >>>>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>> 1.11.3.
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> [1]
> >>>>>>> https://issues.apache.org/jira/browse/FLINK-19698
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> On Wed, Oct 28, 2020 at 12:32 AM Tzu-Li
> >>>> (Gordon)
> >>>>>>> Tai <
> >>>>>>>>>>>>>>>>>>> tzulitai@apache.org>
> >>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> Thanks for the replies so far!
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> Just to provide a brief update on the
> >>> status
> >>>>> of
> >>>>>>>>>> blockers
> >>>>>>>>>>>> for
> >>>>>>>>>>>>>>>>> 1.11.3
> >>>>>>>>>>>>>>>>>> so
> >>>>>>>>>>>>>>>>>>>> far:
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> *PR opened, pending reviewer*-
> >>> [FLINK-19717]
> >>>>>>>>>>>>>>>>>> SourceReaderBase.pollNext
> >>>>>>>>>>>>>>>>>>>> may
> >>>>>>>>>>>>>>>>>>>>> return END_OF_INPUT if SplitReader.fetch
> >>>>> throws
> >>>>>> (
> >>>>>>>>>>>>>>>>>>>>>
> >>> https://github.com/apache/flink/pull/13776)
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> *PR opened, reviewed + close to being
> >>>> merged*
> >>>>>>>>>>>>>>>>>>>>> - [FLINK-19741] Timer service should
> >> skip
> >>>>>>> restoring
> >>>>>>>>>> from
> >>>>>>>>>>>> raw
> >>>>>>>>>>>>>>> keyed
> >>>>>>>>>>>>>>>>>>> stream
> >>>>>>>>>>>>>>>>>>>>> if it isn't the writer (
> >>>>>>>>>>>>>>>>> https://github.com/apache/flink/pull/13761)
> >>>>>>>>>>>>>>>>>>>>> - [FLINK-19748] Raw keyed stream key
> >> group
> >>>>>>> iterator
> >>>>>>>>>>> should
> >>>>>>>>>>>> be
> >>>>>>>>>>>>>>>>>> skipping
> >>>>>>>>>>>>>>>>>>>>> unwritten key groups (
> >>>>>>>>>>>>>>> https://github.com/apache/flink/pull/13772)
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> *Merged*
> >>>>>>>>>>>>>>>>>>>>> - [FLINK-19154] Application mode deletes
> >>> HA
> >>>>> data
> >>>>>>> in
> >>>>>>>>>> case
> >>>>>>>>>>> of
> >>>>>>>>>>>>>>>>> suspended
> >>>>>>>>>>>>>>>>>>>>> ZooKeeper connection
> >>>>>>>>>>>>>>>>>>>>> - [FLINK-19569] Upgrade ICU4J to 67.1+
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> Right now as it seems, progress is
> >> mainly
> >>>>>> blocked
> >>>>>>>>> on a
> >>>>>>>>>>>>> reviewer
> >>>>>>>>>>>>>>>>> for
> >>>>>>>>>>>>>>>>>>>>> FLINK-19717.
> >>>>>>>>>>>>>>>>>>>>> Meanwhile, Xintong is keeping an eye on
> >>> test
> >>>>>>>>>>> instabilities
> >>>>>>>>>>>>> [1]
> >>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>> see
> >>>>>>>>>>>>>>>>>>> if
> >>>>>>>>>>>>>>>>>>>>> there are any fixes that should be
> >> applied
> >>>> to
> >>>>>>>>>>>> `release-1.11`.
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> This is also a reminder, that if there
> >> are
> >>>>> other
> >>>>>>>>>> blockers
> >>>>>>>>>>>>> that
> >>>>>>>>>>>>>>> we
> >>>>>>>>>>>>>>>>>> need
> >>>>>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>>> be aware of, or a need to re-establish
> >>>>> estimated
> >>>>>>>>> time
> >>>>>>>>>> for
> >>>>>>>>>>>>>>> getting
> >>>>>>>>>>>>>>>>>> fixes
> >>>>>>>>>>>>>>>>>>>> in
> >>>>>>>>>>>>>>>>>>>>> and delay the RC for 1.11.3, please do
> >> let
> >>>> us
> >>>>>>> know!
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> Cheers,
> >>>>>>>>>>>>>>>>>>>>> Gordon
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> [1]
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>>
> >>
> https://issues.apache.org/jira/issues/?jql=project%20%3D%20FLINK%20AND%20status%20in%20(Open%2C%20%22In%20Progress%22%2C%20Reopened)%20AND%20affectedVersion%20in%20(1.11.0%2C%201.11.1%2C%201.11.2)%20AND%20labels%20%3D%20test-stability
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> On Mon, Oct 26, 2020 at 9:43 PM Kostas
> >>>>> Kloudas <
> >>>>>>>>>>>>>>>>> kkloudas@apache.org>
> >>>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> +1 for releasing Flink 1.11.3 as it
> >>>>> contains a
> >>>>>>>>> number
> >>>>>>>>>>> of
> >>>>>>>>>>>>>>>>> important
> >>>>>>>>>>>>>>>>>>>>>> fixes and thanks Gordon and Xintong
> >> for
> >>>>>>>>> volunteering.
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> Cheers,
> >>>>>>>>>>>>>>>>>>>>>> Kostas
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> On Mon, Oct 26, 2020 at 4:37 AM Yu Li
> >> <
> >>>>>>>>>>> carp84@gmail.com>
> >>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> +1 for releasing Flink 1.11.3, and
> >>>> thanks
> >>>>>>> Gordon
> >>>>>>>>>> and
> >>>>>>>>>>>>>>> Xintong
> >>>>>>>>>>>>>>>>> for
> >>>>>>>>>>>>>>>>>>>>>>> volunteering as our release
> >> managers.
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> Best Regards,
> >>>>>>>>>>>>>>>>>>>>>>> Yu
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> On Mon, 26 Oct 2020 at 09:45,
> >> Xintong
> >>>>> Song <
> >>>>>>>>>>>>>>>>>> tonysong820@gmail.com>
> >>>>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> Thanks Gordan for starting this
> >>>>>> discussion.
> >>>>>>>>>>>>>>>>>>>>>>>> My pleasure to help with the
> >> release
> >>>>>>> process.
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> Thank you~
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> Xintong Song
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> On Fri, Oct 23, 2020 at 11:29 PM
> >>> Till
> >>>>>>>>> Rohrmann <
> >>>>>>>>>>>>>>>>>>>> trohrmann@apache.org
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> Thanks for starting this
> >>> discussion
> >>>>>>> Gordon.
> >>>>>>>>>> There
> >>>>>>>>>>>> are
> >>>>>>>>>>>>>>> over
> >>>>>>>>>>>>>>>>>> 100
> >>>>>>>>>>>>>>>>>>>>> issues
> >>>>>>>>>>>>>>>>>>>>>>>>> which are fixed for 1.11.3.
> >> Hence
> >>> +1
> >>>>>> for a
> >>>>>>>>>>> soonish
> >>>>>>>>>>>>>>> 1.11.3
> >>>>>>>>>>>>>>>>>>>> release.
> >>>>>>>>>>>>>>>>>>>>>> Thanks
> >>>>>>>>>>>>>>>>>>>>>>>>> for volunteering as our release
> >>>>> managers
> >>>>>>>>> Gordon
> >>>>>>>>>>> and
> >>>>>>>>>>>>>>>>> Xintong!
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> Cheers,
> >>>>>>>>>>>>>>>>>>>>>>>>> Till
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> On Fri, Oct 23, 2020 at 5:02 PM
> >>>> Tzu-Li
> >>>>>>>>> (Gordon)
> >>>>>>>>>>>> Tai <
> >>>>>>>>>>>>>>>>>>>>>> tzulitai@apache.org
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> Hi,
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> Xintong and I would like to
> >>> start a
> >>>>>>>>> discussion
> >>>>>>>>>>> for
> >>>>>>>>>>>>>>>>> releasing
> >>>>>>>>>>>>>>>>>>>> Flink
> >>>>>>>>>>>>>>>>>>>>>>>> 1.11.3
> >>>>>>>>>>>>>>>>>>>>>>>>>> soon.
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> It seems like we already have a
> >>> few
> >>>>>>>>> pressing
> >>>>>>>>>>>> issues
> >>>>>>>>>>>>>>> that
> >>>>>>>>>>>>>>>>>> needs
> >>>>>>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>>> be
> >>>>>>>>>>>>>>>>>>>>>>>>>> included in a new hotfix
> >> release:
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>   - Heap-based timers’ restore
> >>>>>> behaviour
> >>>>>>>>> is
> >>>>>>>>>>>>> causing a
> >>>>>>>>>>>>>>>>>>> critical
> >>>>>>>>>>>>>>>>>>>>>> recovery
> >>>>>>>>>>>>>>>>>>>>>>>>>>   issue for StateFun [1] [2]
> >>> [3].
> >>>>>>>>>>>>>>>>>>>>>>>>>>   - There are several
> >> robustness
> >>>>>> issues
> >>>>>>>>> for
> >>>>>>>>>> the
> >>>>>>>>>>>>>>> FLIP-27
> >>>>>>>>>>>>>>>>> new
> >>>>>>>>>>>>>>>>>>>>> source
> >>>>>>>>>>>>>>>>>>>>>> API,
> >>>>>>>>>>>>>>>>>>>>>>>>>>   such as [4]. We already have
> >>>> some
> >>>>>>> users
> >>>>>>>>>> using
> >>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>> FLIP-27
> >>>>>>>>>>>>>>>>>>> API
> >>>>>>>>>>>>>>>>>>>>>> with
> >>>>>>>>>>>>>>>>>>>>>>>>>> 1.11.x,
> >>>>>>>>>>>>>>>>>>>>>>>>>>   so it would be important to
> >>> get
> >>>>>> those
> >>>>>>>>> fixes
> >>>>>>>>>>> in
> >>>>>>>>>>>>> for
> >>>>>>>>>>>>>>>>> 1.11.x
> >>>>>>>>>>>>>>>>>>> as
> >>>>>>>>>>>>>>>>>>>>>> well.
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> Apart from the issues that are
> >>>>> already
> >>>>>>>>> marked
> >>>>>>>>>> as
> >>>>>>>>>>>>>>> blocker
> >>>>>>>>>>>>>>>>> for
> >>>>>>>>>>>>>>>>>>>>> 1.11.3
> >>>>>>>>>>>>>>>>>>>>>> in
> >>>>>>>>>>>>>>>>>>>>>>>> our
> >>>>>>>>>>>>>>>>>>>>>>>>>> JIRA [5], please let us know in
> >>>> this
> >>>>>>>>> thread if
> >>>>>>>>>>>> there
> >>>>>>>>>>>>>>> is
> >>>>>>>>>>>>>>>>>>> already
> >>>>>>>>>>>>>>>>>>>>>> ongoing
> >>>>>>>>>>>>>>>>>>>>>>>>>> work for other important fixes
> >>> that
> >>>>> we
> >>>>>>>>> should
> >>>>>>>>>>> try
> >>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>> include.
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> Xintong and I would like to
> >>>> volunteer
> >>>>>> for
> >>>>>>>>>>> managing
> >>>>>>>>>>>>>>> this
> >>>>>>>>>>>>>>>>>>> release,
> >>>>>>>>>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>>>>>>>>> will
> >>>>>>>>>>>>>>>>>>>>>>>>>> try to communicate the priority
> >>> of
> >>>>>>> pending
> >>>>>>>>>>>> blockers
> >>>>>>>>>>>>>>> over
> >>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>> next
> >>>>>>>>>>>>>>>>>>>>>> few
> >>>>>>>>>>>>>>>>>>>>>>>>>> days.
> >>>>>>>>>>>>>>>>>>>>>>>>>> Since the aforementioned issues
> >>> are
> >>>>>> quite
> >>>>>>>>>>>> critical,
> >>>>>>>>>>>>>>> we’d
> >>>>>>>>>>>>>>>>>> like
> >>>>>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>>> aim
> >>>>>>>>>>>>>>>>>>>>>>>>>> for a *feature
> >>>>>>>>>>>>>>>>>>>>>>>>>> freeze by the end of next week
> >>>> (Oct.
> >>>>>>> 30th)*
> >>>>>>>>>> and
> >>>>>>>>>>>>> start
> >>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>> release
> >>>>>>>>>>>>>>>>>>>>>> voting
> >>>>>>>>>>>>>>>>>>>>>>>>>> process the week after.
> >>>>>>>>>>>>>>>>>>>>>>>>>> If that is too short of a
> >> notice
> >>>> and
> >>>>>> you
> >>>>>>>>> might
> >>>>>>>>>>>> need
> >>>>>>>>>>>>>>> more
> >>>>>>>>>>>>>>>>>> time,
> >>>>>>>>>>>>>>>>>>>>>> please
> >>>>>>>>>>>>>>>>>>>>>>>> let
> >>>>>>>>>>>>>>>>>>>>>>>>>> us know!
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> Cheers,
> >>>>>>>>>>>>>>>>>>>>>>>>>> Gordon
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> [1]
> >>>>>>>>>>>>> https://issues.apache.org/jira/browse/FLINK-19692
> >>>>>>>>>>>>>>>>>>>>>>>>>> [2]
> >>>>>>>>>>>>> https://issues.apache.org/jira/browse/FLINK-19741
> >>>>>>>>>>>>>>>>>>>>>>>>>> [3]
> >>>>>>>>>>>>> https://issues.apache.org/jira/browse/FLINK-19748
> >>>>>>>>>>>>>>>>>>>>>>>>>> [4]
> >>>>>>>>>>>>> https://issues.apache.org/jira/browse/FLINK-19717
> >>>>>>>>>>>>>>>>>>>>>>>>>> [5]
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>>
> >>
> https://issues.apache.org/jira/issues/?jql=project%20%3D%20FLINK%20AND%20priority%20%3D%20Blocker%20AND%20fixVersion%20%3D%201.11.3
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> --
> >>>>>>>>>>>>>>>>>> Best, Jingsong Lee
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>>
> >>
>
>
>

Re: [DISCUSS] Releasing Apache Flink 1.11.3

Posted by Dian Fu <di...@gmail.com>.
Hi Xintong,

I want to bring one more issue to your attention [1]. The test case UnalignedCheckpointCompatibilityITCase.test failed several times in the last nightly test of release-1.11. We need to figure out if it's just an instable test or caused by recent changes.

[1] https://issues.apache.org/jira/browse/FLINK-20065

> 在 2020年11月10日，上午9:24，Xintong Song <to...@gmail.com> 写道：
> 
> Thanks for the replies.
> 
> Thank you~
> 
> Xintong Song
> 
> 
> 
> On Tue, Nov 10, 2020 at 1:09 AM Becket Qin <be...@gmail.com> wrote:
> 
>> Hi Xintong,
>> 
>> Thanks for driving the release. Just want to sync up on the FLIP-27
>> backporting. Stephan and I are still trying to backport a bunch of patches
>> of Source to 1.11.3. Including:
>> 
>> [FLINK-19698][connector/common] Add a close() method to the SplitReader.
>> [FLINK-19717] SourceReaderBase.pollNext may return END_OF_INPUT if
>> SplitReader.fetch throws
>> [FLINK-19535] [connector/common] Avoid failing a job multiple times in
>> SourceCoordinator.
>> [FLINK-19265] [FLINK-20049][core] Source API final adjustments.
>> 
>> and a few more fixes.
>> 
>> We are currently trying to fix them in 1.12 first so it might take a little
>> longer to backport them to 1.11.3. I think it will probably take us a few
>> more days to finish the backport. So that would roughly be the end of this
>> week.
>> 
>> Thanks,
>> 
>> Jiangjie (Becket) Qin
>> 
>> 
>> 
>> 
>> On Mon, Nov 9, 2020 at 9:57 PM Till Rohrmann <tr...@apache.org> wrote:
>> 
>>> Yes, I've downgraded FLINK-19816 to critical.
>>> 
>>> Cheers,
>>> Till
>>> 
>>> On Mon, Nov 9, 2020 at 10:19 AM Xintong Song <to...@gmail.com>
>>> wrote:
>>> 
>>>> Thanks for the notice, Till.
>>>> 
>>>> I just checked and found FLINK-20033 is already fixed. Shall we also
>>>> downgrade FLINK-19816 to `Critical`?
>>>> 
>>>> Thank you~
>>>> 
>>>> Xintong Song
>>>> 
>>>> 
>>>> 
>>>> On Mon, Nov 9, 2020 at 4:42 PM Till Rohrmann <tr...@apache.org>
>>> wrote:
>>>> 
>>>>> I would like to bring one more critical issue to your attention which
>>> is
>>>>> FLINK-20033 [1]. I believe that this issue is actually causing what
>> has
>>>>> been reported in FLINK-19816 [2]. I hope to have it fixed by the end
>> of
>>>>> today. Once FLINK-20033 is fixed, I think that we don't have to block
>>> the
>>>>> release on FLINK-19816.
>>>>> 
>>>>> [1] https://issues.apache.org/jira/browse/FLINK-20033
>>>>> [2] https://issues.apache.org/jira/browse/FLINK-19816
>>>>> 
>>>>> Cheers,
>>>>> Till
>>>>> 
>>>>> On Mon, Nov 9, 2020 at 4:05 AM Xintong Song <to...@gmail.com>
>>>> wrote:
>>>>> 
>>>>>> Hi devs,
>>>>>> 
>>>>>> I'd like to provide an update on the progress of preparing release
>>>>> 1.11.3.
>>>>>> 
>>>>>> *Blockers*
>>>>>> We currently have 3 remaining blockers. (3 resolved and 1 emerged
>>>>> compared
>>>>>> to last week)
>>>>>> 
>>>>>> - [FLINK-19698] Add close() method and onCheckpointComplete() to
>> the
>>>>>> Source.
>>>>>> The issue has been fixed on the master branch. It's currently
>> blocked
>>>> on
>>>>>> the FLIP-27 backportings to backport it to the 1.11 branch.
>>>>>> 
>>>>>> - [FLINK-19717] SourceReaderBase.pollNext may return END_OF_INPUT
>> if
>>>>>> SplitReader.fetch throws
>>>>>> A PR has been opened and reviewed. From the discussions on the PR,
>> it
>>>>> looks
>>>>>> close to mergeable.
>>>>>> 
>>>>>> - [FLINK-19816] Flink restored from a wrong checkpoint (a very old
>>> one
>>>>> and
>>>>>> not the last completed one)
>>>>>> This is a newly emerged blocker and Matthias is working on it.
>>>>>> 
>>>>>> *Test Instabilities*
>>>>>> We currently have 27 test instabilities[1].
>>>>>> AFAIK, none of them are as serious as to block the 1.11.3 release.
>>>>>> 
>>>>>> *FLIP-27 Backprotings*
>>>>>> 
>>>>>> I noticed that there's no jira issues opened on the FLIP-27
>>> backporting
>>>>>> efforts, which is part of the major efforts planned for the 1.11.3
>>>>> release,
>>>>>> making it hard to track the progress.
>>>>>> 
>>>>>> 
>>>>>> @Stephan and @Becket, could you please share the updates on the
>>>>> backporting
>>>>>> efforts? How is the progress and when are the efforts expected to
>> be
>>>>>> finished? It would be appreciated and helpful if we can have a jira
>>>>> ticket
>>>>>> to track the backporing progress publicly.
>>>>>> 
>>>>>> 
>>>>>> Thank you~
>>>>>> 
>>>>>> Xintong Song
>>>>>> 
>>>>>> 
>>>>>> [1]
>>>>>> 
>>>>>> 
>>>>> 
>>>> 
>>> 
>> https://issues.apache.org/jira/browse/FLINK-16908?filter=12348580&jql=project%20%3D%20FLINK%20AND%20resolution%20%3D%20Unresolved%20AND%20affectedVersion%20in%20(1.11.0%2C%201.11.1%2C%201.11.2%2C%201.11.3)%20AND%20labels%20%3D%20test-stability%20ORDER%20BY%20updated%20DESC%2C%20created%20DESC
>>>>>> 
>>>>>> On Wed, Nov 4, 2020 at 3:58 PM Tzu-Li (Gordon) Tai <
>>>> tzulitai@apache.org>
>>>>>> wrote:
>>>>>> 
>>>>>>> Thanks for the efforts so far with pushing for 1.11.3.
>>>>>>> 
>>>>>>> I'd like to provide a mid-week update on how we're looking with
>> all
>>>> the
>>>>>>> pending blockers and backports:
>>>>>>> 
>>>>>>> *Blockers**:*
>>>>>>> 
>>>>>>> Currently all blockers either have an open PR, or have been
>> merged.
>>>>> I'll
>>>>>>> highlight below the blockers *that still require reviewing
>> efforts
>>>> for
>>>>>> them
>>>>>>> to move forward:*
>>>>>>> 
>>>>>>> - [FLINK-19909] Flink application in attach mode could not
>>> terminate
>>>>> when
>>>>>>> the only job is canceled. PR:
>>>>> https://github.com/apache/flink/pull/13911
>>>>>>> - [FLINK-19717] SourceReaderBase.pollNext may return END_OF_INPUT
>>> if
>>>>>>> SplitReader.fetch throws. PR:
>>>>> https://github.com/apache/flink/pull/13776
>>>>>>> 
>>>>>>> The above PRs currently have no reviews at all yet. They seem to
>>>>> already
>>>>>>> have designated reviewers.
>>>>>>> 
>>>>>>> 
>>>>>>> *Backports:*
>>>>>>> - FLIP-27 robustness improvement backports: Stephan is currently
>>>>> working
>>>>>> on
>>>>>>> backporting several FLIP-27 changes. There are no PRs yet for the
>>>>>>> backported changes.
>>>>>>> - DataStreamUtils.collect() refactorings backport: Steven Wu
>>>> mentioned
>>>>> to
>>>>>>> backport this, but AFAIK this isn't assigned to anyone yet.
>>>>>>> 
>>>>>>> *ETAs*:
>>>>>>> 
>>>>>>> I'd like to request ETAs for the remaining backports, to prevent
>> a
>>>>> creep
>>>>>> in
>>>>>>> the scope of this bugfix release.
>>>>>>> 
>>>>>>> We already have Flink users that would benefit from fixes that
>> have
>>>>> been
>>>>>>> merged to release-1.11 already, so technically speaking the
>>> backports
>>>>>>> should be considered "nice-to-have" (to the best of my knowledge
>> of
>>>> the
>>>>>>> changes) and could potentially be moved to a follow-up 1.11.4.
>>>>>>> Most notably, the Stateful Functions project is already waiting
>> on
>>>>> Flink
>>>>>>> 1.11.3 to address critical recovery issues (please see the
>> StateFun
>>>>> 2.2.1
>>>>>>> release discussion thread [1]).
>>>>>>> 
>>>>>>> @Stephan Ewen <se...@apache.org> @Becket Qin <
>> becket.qin@gmail.com
>>>> 
>>>>>> could
>>>>>>> you provide an ETA for the FLIP-27 backports? It would help to
>> get
>>> a
>>>>>> better
>>>>>>> estimate to decide how we proceed here.
>>>>>>> 
>>>>>>> Cheers,
>>>>>>> Gordon
>>>>>>> 
>>>>>>> [1]
>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>> 
>>>> 
>>> 
>> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Releasing-StateFun-hotfix-version-2-2-1-td46239.html
>>>>>>> 
>>>>>>> On Wed, Nov 4, 2020 at 3:16 PM Tzu-Li (Gordon) Tai <
>>>>> tzulitai@apache.org>
>>>>>>> wrote:
>>>>>>> 
>>>>>>>>> The collect() utils can be picked back, I see no issue with
>>> that
>>>>> (it
>>>>>> is
>>>>>>>> isolated utilities).
>>>>>>>> 
>>>>>>>> Just checking on all the requested backports mentioned in this
>>>>> thread,
>>>>>>> and
>>>>>>>> figuring out which ones seem to still be unassigned / open.
>>>>>>>> 
>>>>>>>> Is someone working on backporting
>>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>> 
>>>> 
>>> 
>> https://github.com/apache/flink/commit/09a7a66b7313fea64817fe960a8da1265b428efc
>>>>>>>> to release-1.11 at the moment?
>>>>>>>> 
>>>>>>>> On Wed, Nov 4, 2020 at 1:39 AM Steven Wu <stevenz3wu@gmail.com
>>> 
>>>>> wrote:
>>>>>>>> 
>>>>>>>>> @Stephan Ewen <se...@apache.org> yeah, we can do that. don't
>>>> worry
>>>>>>> about
>>>>>>>>> it. your earlier email had the perfect explanation on why file
>>>>> source
>>>>>>>>> shouldn't be backported.
>>>>>>>>> 
>>>>>>>>> On Tue, Nov 3, 2020 at 3:37 AM Stephan Ewen <sewen@apache.org
>>> 
>>>>> wrote:
>>>>>>>>> 
>>>>>>>>>> @Steven would it be possible to initially copy some of the
>>> code
>>>>> into
>>>>>>> the
>>>>>>>>>> iceberg source and later replace it by a dependency on the
>>> Flink
>>>>>> file
>>>>>>>>>> source?
>>>>>>>>>> 
>>>>>>>>>> On Mon, Nov 2, 2020 at 8:33 PM Steven Wu <
>>> stevenz3wu@gmail.com>
>>>>>>> wrote:
>>>>>>>>>> 
>>>>>>>>>>> Stephan, thanks a lot for explaining the file connector.
>>> that
>>>>>> makes
>>>>>>>>>> sense.
>>>>>>>>>>> 
>>>>>>>>>>> I was asking because we were trying to reuse some of the
>>>>>>>>> implementations
>>>>>>>>>> in
>>>>>>>>>>> the file source for Iceberg source. Flink Iceberg source
>>> lives
>>>>> in
>>>>>>> the
>>>>>>>>>>> Iceberg repo, which is not possible to code against the
>>> master
>>>>>>> branch
>>>>>>>>> of
>>>>>>>>>>> the Flink code.
>>>>>>>>>>> 
>>>>>>>>>>> On Mon, Nov 2, 2020 at 3:31 AM Stephan Ewen <
>>> sewen@apache.org
>>>>> 
>>>>>>> wrote:
>>>>>>>>>>> 
>>>>>>>>>>>> Hi Steven!
>>>>>>>>>>>> 
>>>>>>>>>>>> So far there are no plans to pick back the file system
>>>>> connector
>>>>>>>>> code.
>>>>>>>>>>> This
>>>>>>>>>>>> is still evolving and not finalized for 1.12, so I don't
>>>> feel
>>>>> it
>>>>>>> is
>>>>>>>>> a
>>>>>>>>>>> good
>>>>>>>>>>>> candidate to be backported.
>>>>>>>>>>>> However, with the base connector changes backported, you
>>>>> should
>>>>>> be
>>>>>>>>> able
>>>>>>>>>>> to
>>>>>>>>>>>> run the file connector code from master against 1.11.3.
>>>>>>>>>>>> 
>>>>>>>>>>>> The collect() utils can be picked back, I see no issue
>>> with
>>>>> that
>>>>>>>>> (it is
>>>>>>>>>>>> isolated utilities).
>>>>>>>>>>>> 
>>>>>>>>>>>> Best,
>>>>>>>>>>>> Stephan
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> On Mon, Nov 2, 2020 at 3:02 AM Steven Wu <
>>>>> stevenz3wu@gmail.com>
>>>>>>>>> wrote:
>>>>>>>>>>>> 
>>>>>>>>>>>>> Basically, it would be great to get the latest code in
>>> the
>>>>>>>>>>>>> flink-connector-files (FLIP-27).
>>>>>>>>>>>>> 
>>>>>>>>>>>>> On Sat, Oct 31, 2020 at 9:57 AM Steven Wu <
>>>>>> stevenz3wu@gmail.com
>>>>>>>> 
>>>>>>>>>>> wrote:
>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Stephan, it will be great if we can also backport
>> the
>>>>>>>>>> DataStreamUtils
>>>>>>>>>>>>>> related commits that help with collecting output
>> from
>>>>>>> unbounded
>>>>>>>>>>>> streams.
>>>>>>>>>>>>>> e.g.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>> 
>>>> 
>>> 
>> https://github.com/apache/flink/commit/09a7a66b7313fea64817fe960a8da1265b428efc
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> I tried to copy and paste the code to unblock
>> myself.
>>>> but
>>>>> it
>>>>>>>>>> quickly
>>>>>>>>>>>> got
>>>>>>>>>>>>>> into the rabbit hole of more and more code.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> On Fri, Oct 30, 2020 at 11:02 AM Stephan Ewen <
>>>>>>> sewen@apache.org
>>>>>>>>>> 
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> I have started with backporting the source API
>>> changes.
>>>>>> Some
>>>>>>>>> minor
>>>>>>>>>>>>>>> conflicts to solve, will need a bit more to finish
>>>> this.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> On Fri, Oct 30, 2020 at 7:25 AM Tzu-Li (Gordon)
>> Tai <
>>>>>>>>>>>>> tzulitai@apache.org>
>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> @Stephan Ewen <se...@apache.org>
>>>>>>>>>>>>>>>> Are there already plans or ongoing efforts for
>>>>>> backporting
>>>>>>>>> the
>>>>>>>>>>> list
>>>>>>>>>>>> of
>>>>>>>>>>>>>>>> FLIP-27 changes that you posted?
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> On Thu, Oct 29, 2020 at 7:08 PM Xintong Song <
>>>>>>>>>>> tonysong820@gmail.com
>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Hi folks,
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Just to provide some updates concerning the
>> status
>>>> on
>>>>>> the
>>>>>>>>>>>>>>>>> test instabilities.
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Currently, we have 30 unresolved tickets labeled
>>>> with
>>>>>>>>> `Affects
>>>>>>>>>>>>> Version`
>>>>>>>>>>>>>>>>> 1.11.x.
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>> 
>>>> 
>>> 
>> https://issues.apache.org/jira/browse/FLINK-19775?filter=12348580&jql=project%20%3D%20FLINK%20AND%20resolution%20%3D%20Unresolved%20AND%20affectedVersion%20in%20(1.11.0%2C%201.11.1%2C%201.11.2%2C%201.11.3)%20AND%20labels%20%3D%20test-stability%20ORDER%20BY%20created%20DESC
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Among the 30 tickets, 11 of them are:
>>>>>>>>>>>>>>>>> - Have occured in the recent 3 months
>>>>>>>>>>>>>>>>> - Not confirmed to be pure testability issues
>>>>>>>>>>>>>>>>> - Not confirmed to be rare condition cases
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> It would be nice if someone familiar with these
>>>>>> components
>>>>>>>>> can
>>>>>>>>>>>> take a
>>>>>>>>>>>>>>> look
>>>>>>>>>>>>>>>>> into these issues.
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> -
>>> https://issues.apache.org/jira/browse/FLINK-17159
>>>>>> (ES6)
>>>>>>>>>>>>>>>>> -
>>> https://issues.apache.org/jira/browse/FLINK-17912
>>>>>>> (Kafka)
>>>>>>>>>>>>>>>>> -
>>> https://issues.apache.org/jira/browse/FLINK-17949
>>>>>>> (Kafka)
>>>>>>>>>>>>>>>>> ⁃
>>> https://issues.apache.org/jira/browse/FLINK-18444
>>>>>>> (Kafka)
>>>>>>>>>>>>>>>>> -
>>> https://issues.apache.org/jira/browse/FLINK-18634
>>>>>>> (Kafka)
>>>>>>>>>>>>>>>>> -
>>> https://issues.apache.org/jira/browse/FLINK-18648
>>>>>>> (Kafka)
>>>>>>>>>>>>>>>>> -
>>> https://issues.apache.org/jira/browse/FLINK-18807
>>>>>>> (Kafka)
>>>>>>>>>>>>>>>>> -
>>> https://issues.apache.org/jira/browse/FLINK-19369
>>>>>>>>>>>> (BlobClientTest)
>>>>>>>>>>>>>>>>> -
>>> https://issues.apache.org/jira/browse/FLINK-19436
>>>>>>> (TPCDS)
>>>>>>>>>>>>>>>>> -
>>> https://issues.apache.org/jira/browse/FLINK-19690
>>>>>>>>>>>> (Format/Parquet)
>>>>>>>>>>>>>>>>> -
>>> https://issues.apache.org/jira/browse/FLINK-19775
>>>>>>>>>>>>>>>>> (SystemProcessingTimeServiceTest)
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Thank you~
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Xintong Song
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> On Thu, Oct 29, 2020 at 10:21 AM Jingsong Li <
>>>>>>>>>>>> jingsonglee0@gmail.com
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> +1 to backport the FLIP-27 adjustments to
>>> 1.11.x.
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> If possible, that would be great. Many people
>>> are
>>>>>>> looking
>>>>>>>>>>> forward
>>>>>>>>>>>>> to
>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>> FLIP-27 interface, but they don't want to take
>>> the
>>>>>> risk
>>>>>>> to
>>>>>>>>>>>> upgrade
>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>> 1.12
>>>>>>>>>>>>>>>>>> (And wait 1.12). After all, 1.11 is a
>> relatively
>>>>>> stable
>>>>>>>>>>> version.
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>>>>> Jingsong
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> On Thu, Oct 29, 2020 at 1:24 AM Stephan Ewen <
>>>>>>>>>> sewen@apache.org
>>>>>>>>>>>> 
>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> Thanks for starting this.
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> +1 form my side to backport the FLIP-27
>>>>> adjustments
>>>>>> to
>>>>>>>>>>> 1.11.x.
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> There were quite a few changes, and I think
>> we
>>>>> need
>>>>>> to
>>>>>>>>>>>>> cherry-pick
>>>>>>>>>>>>>>>>> them
>>>>>>>>>>>>>>>>>> all
>>>>>>>>>>>>>>>>>>> to not get some inconsistent mix of changes
>>> and
>>>>> many
>>>>>>>>> merge
>>>>>>>>>>>>>>> conflicts.
>>>>>>>>>>>>>>>>>>> I made a list below of what we need to add
>> to
>>>>>>>>>> "release-1.11".
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> *#### Core Source API Changes to backport
>> (in
>>>>>> REVERSE
>>>>>>>>>> order)*
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>  (Use: "git log
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>> flink-core/src/main/java/org/apache/flink/api/connector/source")
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> commit
>>> 162c072e9265a7b6dd9d6f5459eb7974091c4c4e
>>>>>>>>>>>>>>>>>>> [FLINK-19492][core] Consolidate Source
>> Events
>>>>>> between
>>>>>>>>>> Source
>>>>>>>>>>>> API
>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>> Split
>>>>>>>>>>>>>>>>>>> Reader API
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> commit
>>> 428fbbad85379f71a9fbf1a1ab3e3dcfa9006436
>>>>>>>>>>>>>>>>>>> [FLINK-19205][core] Add access to
>>> configuration
>>>>> and
>>>>>>>>>> hostname
>>>>>>>>>>> in
>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>> SourceReaderContext
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> commit
>>> 723e1790744ddc78d19c7c978442af1383f38d33
>>>>>>>>>>>>>>>>>>> [hotfix][core] Add to Source Enumerator
>>>>> convenience
>>>>>>>>> methods
>>>>>>>>>>> to
>>>>>>>>>>>>>>> assign
>>>>>>>>>>>>>>>>>>> single split
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> *#### Connector Base Changes to Backport (in
>>>>> REVERSE
>>>>>>>>>> order)*
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>  (Use: "git log
>>>>>>> flink-connectors/flink-connector-base")
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> commit
>>> 401f56fe9d6b0271260edf9787cdcbfe4d03874d
>>>>>>>>>>>>>>>>>>> [FLINK-19427][FLINK-19489][tests] Fix test
>>>>>> conditions
>>>>>>>>> for
>>>>>>>>>>>>>>>>>>> 
>>>>>>> 'SplitFetcherTest.testNotifiesWhenGoingIdleConcurrent()'
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> commit
>>> 68c5c2ff779d82a1ff81ffaf60d8a1b283797db1
>>>>>>>>>>>>>>>>>>> [FLINK-19448][connector base] Explicitly
>> check
>>>> for
>>>>>>>>>>> un-expected
>>>>>>>>>>>>>>>>> condition
>>>>>>>>>>>>>>>>>>> that would leave an inconsistent state
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> commit
>>> 162c072e9265a7b6dd9d6f5459eb7974091c4c4e
>>>>>>>>>>>>>>>>>>> [FLINK-19492][core] Consolidate Source
>> Events
>>>>>> between
>>>>>>>>>> Source
>>>>>>>>>>>> API
>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>> Split
>>>>>>>>>>>>>>>>>>> Reader API
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> commit
>>> c1ca7a4c7c21ec8868c14cdddd43c559625b794c
>>>>>>>>>>>>>>>>>>> [refactor][tests] Move some source test
>> utils
>>>> from
>>>>>>>>>>>>>>>>> flink-connector-base
>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>> flink-core
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> commit
>>> ee5c4c211c35c70d28252363bbc8400453609977
>>>>>>>>>>>>>>>>>>> [FLINK-19251][connectors] Avoid confusing
>>> queue
>>>>>>>>> handling in
>>>>>>>>>>>>>>>>>>> "SplitReader.handleSplitsChanges()"
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> commit
>>> 5abef56b2bf85bcac786f6b16b6899b6cced7176
>>>>>>>>>>>>>>>>>>> [FLINK-19250][connectors] Fix error
>>> propagation
>>>> in
>>>>>>>>>> connector
>>>>>>>>>>>> base
>>>>>>>>>>>>>>>>>>> (SplitFetcherManager).
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> commit
>>> 8fcca837c55a9216595ee4c03038b52747098dbb
>>>>>>>>>>>>>>>>>>> [hotfix][connectors] Improve JavaDocs for
>>>>>>>>>>>>>>> SingleThreadFetcherManager
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> commit
>>> 4700bb5dde3303cbe98882f6beb7379425717b01
>>>>>>>>>>>>>>>>>>> [FLINK-19225][connectors] Various small
>>>>> improvements
>>>>>>> to
>>>>>>>>>>>>>>>>> SourceReaderBase
>>>>>>>>>>>>>>>>>>> (part 2)
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> commit
>>> 12261c6b7ed6478a9b9f6a69cb58246b83cab9b7
>>>>>>>>>>>>>>>>>>> [FLINK-17393][connectors] (follow-up) Wakeup
>>> the
>>>>>>>>>>> SplitFetchers
>>>>>>>>>>>>> more
>>>>>>>>>>>>>>>>>>> elegantly.
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> commit
>>> c60aaff0249bfd6b5871b7f82e03efc487a54d6b
>>>>>>>>>>>>>>>>>>> [hotfix][tests] Extend test coverage for
>>>>>>>>>>>>>>> FutureCompletingBlockingQueue
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> commit
>>> cef8a587d7fd2fe64cc644da5ed095d82e46f631
>>>>>>>>>>>>>>>>>>> [FLINK-19245][connectors] Set default
>> capacity
>>>> for
>>>>>>>>>>>>>>>>>>> FutureCompletingBlockingQueue.
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> commit
>>> 4ea95782b4c6a2538153d4d16ad3f4839c7de0fb
>>>>>>>>>>>>>>>>>>> [FLINK-19223][connectors] Simplify
>>> Availability
>>>>>> Future
>>>>>>>>>> Model
>>>>>>>>>>> in
>>>>>>>>>>>>>>> Base
>>>>>>>>>>>>>>>>>>> Connector
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> commit
>>> 511857049ba30c8ff0ee56da551fa4a479dc583e
>>>>>>>>>>>>>>>>>>> [FLINK-18128][connectors] Ensure idle split
>>>>> fetchers
>>>>>>>>> lead
>>>>>>>>>> to
>>>>>>>>>>>>>>>>> availability
>>>>>>>>>>>>>>>>>>> notifications.
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> commit
>>> a8206467af0830dcb89623ea068b5ca3b3450c92
>>>>>>>>>>>>>>>>>>> [refactor][core] Eagerly initialize the
>>>> FetchTask
>>>>> to
>>>>>>>>>> support
>>>>>>>>>>>>> proper
>>>>>>>>>>>>>>>>> unit
>>>>>>>>>>>>>>>>>>> testing
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> commit
>>> 3b2f54bcb437f98e6137c904045cc51072b5c06b
>>>>>>>>>>>>>>>>>>> [hotfix][tests] Move constants in
>>>> SplitFetcherTest
>>>>>>>>> relevant
>>>>>>>>>>> to
>>>>>>>>>>>>> only
>>>>>>>>>>>>>>>>> one
>>>>>>>>>>>>>>>>>>> test into test method
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> commit
>>> d7625760a75a508bf05bcddc380bb4d62ee1743e
>>>>>>>>>>>>>>>>>>> [FLINK-19225][connectors] Various small
>>>>> improvements
>>>>>>> to
>>>>>>>>>>>>>>>>> SourceReaderBase
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> commit
>>> a5b0d3297748c1be47ad579a88f24df2255a8df1
>>>>>>>>>>>>>>>>>>> [FLINK-17393][connectors] Wakeup the
>>>> SplitFetchers
>>>>>>> more
>>>>>>>>>>>>> elegantly.
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> commit
>>> f42a3ebc3e81a034b7221a803c153636fef34903
>>>>>>>>>>>>>>>>>>> [FLINK-18680][connectors] Make connector
>> base
>>>>>>>>>>>> RecordsWithSplitIds
>>>>>>>>>>>>>>> more
>>>>>>>>>>>>>>>>>>> lightweight.
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> commit
>>> e3d273de822b085183d09b275a445879ff94b350
>>>>>>>>>>>>>>>>>>> [FLINK-19162][connectors] Add 'recycle()' to
>>> the
>>>>>>>>>>>>>>> RecordsWithSplitIds
>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>> support reuse of heavy objects.
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> commit
>>> 8ebc464c2520453a70001cd712abc8dee6ee89e0
>>>>>>>>>>>>>>>>>>> [hotfix][testing] Add a set of
>> parameterizable
>>>>>> testing
>>>>>>>>>> mocks
>>>>>>>>>>>> for
>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>> Split
>>>>>>>>>>>>>>>>>>> Reader API
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> commit
>>> 930a07438be1185388d7150640f294dfe2a5d378
>>>>>>>>>>>>>>>>>>> [hotfix][connectors] Add RequestSplitEvent
>> to
>>>>>>>>>>>>>>> 'flink-connector-base'
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> commit
>>> e945ce8a933bc378844782f784ca473c767ca159
>>>>>>>>>>>>>>>>>>> [refactor][connectors] Backport of the
>>>>>> connector-base
>>>>>>>>>>> exception
>>>>>>>>>>>>>>>>> handling
>>>>>>>>>>>>>>>>>>> from the Kafka Connector Pull Request
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> On Wed, Oct 28, 2020 at 5:49 PM Steven Wu <
>>>>>>>>>>>> stevenz3wu@gmail.com>
>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> I would love to see this FLIP-27 source
>>>>> interface
>>>>>>>>>>> improvement
>>>>>>>>>>>>> [1]
>>>>>>>>>>>>>>>>> made
>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>> 1.11.3.
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> [1]
>>>>>>> https://issues.apache.org/jira/browse/FLINK-19698
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> On Wed, Oct 28, 2020 at 12:32 AM Tzu-Li
>>>> (Gordon)
>>>>>>> Tai <
>>>>>>>>>>>>>>>>>>> tzulitai@apache.org>
>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> Thanks for the replies so far!
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> Just to provide a brief update on the
>>> status
>>>>> of
>>>>>>>>>> blockers
>>>>>>>>>>>> for
>>>>>>>>>>>>>>>>> 1.11.3
>>>>>>>>>>>>>>>>>> so
>>>>>>>>>>>>>>>>>>>> far:
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> *PR opened, pending reviewer*-
>>> [FLINK-19717]
>>>>>>>>>>>>>>>>>> SourceReaderBase.pollNext
>>>>>>>>>>>>>>>>>>>> may
>>>>>>>>>>>>>>>>>>>>> return END_OF_INPUT if SplitReader.fetch
>>>>> throws
>>>>>> (
>>>>>>>>>>>>>>>>>>>>> 
>>> https://github.com/apache/flink/pull/13776)
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> *PR opened, reviewed + close to being
>>>> merged*
>>>>>>>>>>>>>>>>>>>>> - [FLINK-19741] Timer service should
>> skip
>>>>>>> restoring
>>>>>>>>>> from
>>>>>>>>>>>> raw
>>>>>>>>>>>>>>> keyed
>>>>>>>>>>>>>>>>>>> stream
>>>>>>>>>>>>>>>>>>>>> if it isn't the writer (
>>>>>>>>>>>>>>>>> https://github.com/apache/flink/pull/13761)
>>>>>>>>>>>>>>>>>>>>> - [FLINK-19748] Raw keyed stream key
>> group
>>>>>>> iterator
>>>>>>>>>>> should
>>>>>>>>>>>> be
>>>>>>>>>>>>>>>>>> skipping
>>>>>>>>>>>>>>>>>>>>> unwritten key groups (
>>>>>>>>>>>>>>> https://github.com/apache/flink/pull/13772)
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> *Merged*
>>>>>>>>>>>>>>>>>>>>> - [FLINK-19154] Application mode deletes
>>> HA
>>>>> data
>>>>>>> in
>>>>>>>>>> case
>>>>>>>>>>> of
>>>>>>>>>>>>>>>>> suspended
>>>>>>>>>>>>>>>>>>>>> ZooKeeper connection
>>>>>>>>>>>>>>>>>>>>> - [FLINK-19569] Upgrade ICU4J to 67.1+
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> Right now as it seems, progress is
>> mainly
>>>>>> blocked
>>>>>>>>> on a
>>>>>>>>>>>>> reviewer
>>>>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>>>>>>> FLINK-19717.
>>>>>>>>>>>>>>>>>>>>> Meanwhile, Xintong is keeping an eye on
>>> test
>>>>>>>>>>> instabilities
>>>>>>>>>>>>> [1]
>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>> see
>>>>>>>>>>>>>>>>>>> if
>>>>>>>>>>>>>>>>>>>>> there are any fixes that should be
>> applied
>>>> to
>>>>>>>>>>>> `release-1.11`.
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> This is also a reminder, that if there
>> are
>>>>> other
>>>>>>>>>> blockers
>>>>>>>>>>>>> that
>>>>>>>>>>>>>>> we
>>>>>>>>>>>>>>>>>> need
>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>> be aware of, or a need to re-establish
>>>>> estimated
>>>>>>>>> time
>>>>>>>>>> for
>>>>>>>>>>>>>>> getting
>>>>>>>>>>>>>>>>>> fixes
>>>>>>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>>>>>> and delay the RC for 1.11.3, please do
>> let
>>>> us
>>>>>>> know!
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>>>>>>>>> Gordon
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> [1]
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>> 
>>>> 
>>> 
>> https://issues.apache.org/jira/issues/?jql=project%20%3D%20FLINK%20AND%20status%20in%20(Open%2C%20%22In%20Progress%22%2C%20Reopened)%20AND%20affectedVersion%20in%20(1.11.0%2C%201.11.1%2C%201.11.2)%20AND%20labels%20%3D%20test-stability
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> On Mon, Oct 26, 2020 at 9:43 PM Kostas
>>>>> Kloudas <
>>>>>>>>>>>>>>>>> kkloudas@apache.org>
>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> +1 for releasing Flink 1.11.3 as it
>>>>> contains a
>>>>>>>>> number
>>>>>>>>>>> of
>>>>>>>>>>>>>>>>> important
>>>>>>>>>>>>>>>>>>>>>> fixes and thanks Gordon and Xintong
>> for
>>>>>>>>> volunteering.
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>>>>>>>>>> Kostas
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> On Mon, Oct 26, 2020 at 4:37 AM Yu Li
>> <
>>>>>>>>>>> carp84@gmail.com>
>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> +1 for releasing Flink 1.11.3, and
>>>> thanks
>>>>>>> Gordon
>>>>>>>>>> and
>>>>>>>>>>>>>>> Xintong
>>>>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>>>>>>>>> volunteering as our release
>> managers.
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>>>>>>>>>>>>> Yu
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> On Mon, 26 Oct 2020 at 09:45,
>> Xintong
>>>>> Song <
>>>>>>>>>>>>>>>>>> tonysong820@gmail.com>
>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> Thanks Gordan for starting this
>>>>>> discussion.
>>>>>>>>>>>>>>>>>>>>>>>> My pleasure to help with the
>> release
>>>>>>> process.
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> Thank you~
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> Xintong Song
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> On Fri, Oct 23, 2020 at 11:29 PM
>>> Till
>>>>>>>>> Rohrmann <
>>>>>>>>>>>>>>>>>>>> trohrmann@apache.org
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for starting this
>>> discussion
>>>>>>> Gordon.
>>>>>>>>>> There
>>>>>>>>>>>> are
>>>>>>>>>>>>>>> over
>>>>>>>>>>>>>>>>>> 100
>>>>>>>>>>>>>>>>>>>>> issues
>>>>>>>>>>>>>>>>>>>>>>>>> which are fixed for 1.11.3.
>> Hence
>>> +1
>>>>>> for a
>>>>>>>>>>> soonish
>>>>>>>>>>>>>>> 1.11.3
>>>>>>>>>>>>>>>>>>>> release.
>>>>>>>>>>>>>>>>>>>>>> Thanks
>>>>>>>>>>>>>>>>>>>>>>>>> for volunteering as our release
>>>>> managers
>>>>>>>>> Gordon
>>>>>>>>>>> and
>>>>>>>>>>>>>>>>> Xintong!
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>>>>>>>>>>>>> Till
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> On Fri, Oct 23, 2020 at 5:02 PM
>>>> Tzu-Li
>>>>>>>>> (Gordon)
>>>>>>>>>>>> Tai <
>>>>>>>>>>>>>>>>>>>>>> tzulitai@apache.org
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> Xintong and I would like to
>>> start a
>>>>>>>>> discussion
>>>>>>>>>>> for
>>>>>>>>>>>>>>>>> releasing
>>>>>>>>>>>>>>>>>>>> Flink
>>>>>>>>>>>>>>>>>>>>>>>> 1.11.3
>>>>>>>>>>>>>>>>>>>>>>>>>> soon.
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> It seems like we already have a
>>> few
>>>>>>>>> pressing
>>>>>>>>>>>> issues
>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>> needs
>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>> be
>>>>>>>>>>>>>>>>>>>>>>>>>> included in a new hotfix
>> release:
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>   - Heap-based timers’ restore
>>>>>> behaviour
>>>>>>>>> is
>>>>>>>>>>>>> causing a
>>>>>>>>>>>>>>>>>>> critical
>>>>>>>>>>>>>>>>>>>>>> recovery
>>>>>>>>>>>>>>>>>>>>>>>>>>   issue for StateFun [1] [2]
>>> [3].
>>>>>>>>>>>>>>>>>>>>>>>>>>   - There are several
>> robustness
>>>>>> issues
>>>>>>>>> for
>>>>>>>>>> the
>>>>>>>>>>>>>>> FLIP-27
>>>>>>>>>>>>>>>>> new
>>>>>>>>>>>>>>>>>>>>> source
>>>>>>>>>>>>>>>>>>>>>> API,
>>>>>>>>>>>>>>>>>>>>>>>>>>   such as [4]. We already have
>>>> some
>>>>>>> users
>>>>>>>>>> using
>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>> FLIP-27
>>>>>>>>>>>>>>>>>>> API
>>>>>>>>>>>>>>>>>>>>>> with
>>>>>>>>>>>>>>>>>>>>>>>>>> 1.11.x,
>>>>>>>>>>>>>>>>>>>>>>>>>>   so it would be important to
>>> get
>>>>>> those
>>>>>>>>> fixes
>>>>>>>>>>> in
>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>>> 1.11.x
>>>>>>>>>>>>>>>>>>> as
>>>>>>>>>>>>>>>>>>>>>> well.
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> Apart from the issues that are
>>>>> already
>>>>>>>>> marked
>>>>>>>>>> as
>>>>>>>>>>>>>>> blocker
>>>>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>>>>>>> 1.11.3
>>>>>>>>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>>>>>>>>> our
>>>>>>>>>>>>>>>>>>>>>>>>>> JIRA [5], please let us know in
>>>> this
>>>>>>>>> thread if
>>>>>>>>>>>> there
>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>> already
>>>>>>>>>>>>>>>>>>>>>> ongoing
>>>>>>>>>>>>>>>>>>>>>>>>>> work for other important fixes
>>> that
>>>>> we
>>>>>>>>> should
>>>>>>>>>>> try
>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>> include.
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> Xintong and I would like to
>>>> volunteer
>>>>>> for
>>>>>>>>>>> managing
>>>>>>>>>>>>>>> this
>>>>>>>>>>>>>>>>>>> release,
>>>>>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>>>>>> will
>>>>>>>>>>>>>>>>>>>>>>>>>> try to communicate the priority
>>> of
>>>>>>> pending
>>>>>>>>>>>> blockers
>>>>>>>>>>>>>>> over
>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>> next
>>>>>>>>>>>>>>>>>>>>>> few
>>>>>>>>>>>>>>>>>>>>>>>>>> days.
>>>>>>>>>>>>>>>>>>>>>>>>>> Since the aforementioned issues
>>> are
>>>>>> quite
>>>>>>>>>>>> critical,
>>>>>>>>>>>>>>> we’d
>>>>>>>>>>>>>>>>>> like
>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>> aim
>>>>>>>>>>>>>>>>>>>>>>>>>> for a *feature
>>>>>>>>>>>>>>>>>>>>>>>>>> freeze by the end of next week
>>>> (Oct.
>>>>>>> 30th)*
>>>>>>>>>> and
>>>>>>>>>>>>> start
>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>> release
>>>>>>>>>>>>>>>>>>>>>> voting
>>>>>>>>>>>>>>>>>>>>>>>>>> process the week after.
>>>>>>>>>>>>>>>>>>>>>>>>>> If that is too short of a
>> notice
>>>> and
>>>>>> you
>>>>>>>>> might
>>>>>>>>>>>> need
>>>>>>>>>>>>>>> more
>>>>>>>>>>>>>>>>>> time,
>>>>>>>>>>>>>>>>>>>>>> please
>>>>>>>>>>>>>>>>>>>>>>>> let
>>>>>>>>>>>>>>>>>>>>>>>>>> us know!
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>>>>>>>>>>>>>> Gordon
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> [1]
>>>>>>>>>>>>> https://issues.apache.org/jira/browse/FLINK-19692
>>>>>>>>>>>>>>>>>>>>>>>>>> [2]
>>>>>>>>>>>>> https://issues.apache.org/jira/browse/FLINK-19741
>>>>>>>>>>>>>>>>>>>>>>>>>> [3]
>>>>>>>>>>>>> https://issues.apache.org/jira/browse/FLINK-19748
>>>>>>>>>>>>>>>>>>>>>>>>>> [4]
>>>>>>>>>>>>> https://issues.apache.org/jira/browse/FLINK-19717
>>>>>>>>>>>>>>>>>>>>>>>>>> [5]
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>> 
>>>> 
>>> 
>> https://issues.apache.org/jira/issues/?jql=project%20%3D%20FLINK%20AND%20priority%20%3D%20Blocker%20AND%20fixVersion%20%3D%201.11.3
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>> Best, Jingsong Lee
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>> 
>>>> 
>>> 
>>

Re: [DISCUSS] Releasing Apache Flink 1.11.3

Posted by Xintong Song <to...@gmail.com>.
Thanks for the replies.

Thank you~

Xintong Song



On Tue, Nov 10, 2020 at 1:09 AM Becket Qin <be...@gmail.com> wrote:

> Hi Xintong,
>
> Thanks for driving the release. Just want to sync up on the FLIP-27
> backporting. Stephan and I are still trying to backport a bunch of patches
> of Source to 1.11.3. Including:
>
> [FLINK-19698][connector/common] Add a close() method to the SplitReader.
> [FLINK-19717] SourceReaderBase.pollNext may return END_OF_INPUT if
> SplitReader.fetch throws
> [FLINK-19535] [connector/common] Avoid failing a job multiple times in
> SourceCoordinator.
> [FLINK-19265] [FLINK-20049][core] Source API final adjustments.
>
> and a few more fixes.
>
> We are currently trying to fix them in 1.12 first so it might take a little
> longer to backport them to 1.11.3. I think it will probably take us a few
> more days to finish the backport. So that would roughly be the end of this
> week.
>
> Thanks,
>
> Jiangjie (Becket) Qin
>
>
>
>
> On Mon, Nov 9, 2020 at 9:57 PM Till Rohrmann <tr...@apache.org> wrote:
>
> > Yes, I've downgraded FLINK-19816 to critical.
> >
> > Cheers,
> > Till
> >
> > On Mon, Nov 9, 2020 at 10:19 AM Xintong Song <to...@gmail.com>
> > wrote:
> >
> > > Thanks for the notice, Till.
> > >
> > > I just checked and found FLINK-20033 is already fixed. Shall we also
> > > downgrade FLINK-19816 to `Critical`?
> > >
> > > Thank you~
> > >
> > > Xintong Song
> > >
> > >
> > >
> > > On Mon, Nov 9, 2020 at 4:42 PM Till Rohrmann <tr...@apache.org>
> > wrote:
> > >
> > > > I would like to bring one more critical issue to your attention which
> > is
> > > > FLINK-20033 [1]. I believe that this issue is actually causing what
> has
> > > > been reported in FLINK-19816 [2]. I hope to have it fixed by the end
> of
> > > > today. Once FLINK-20033 is fixed, I think that we don't have to block
> > the
> > > > release on FLINK-19816.
> > > >
> > > > [1] https://issues.apache.org/jira/browse/FLINK-20033
> > > > [2] https://issues.apache.org/jira/browse/FLINK-19816
> > > >
> > > > Cheers,
> > > > Till
> > > >
> > > > On Mon, Nov 9, 2020 at 4:05 AM Xintong Song <to...@gmail.com>
> > > wrote:
> > > >
> > > > > Hi devs,
> > > > >
> > > > > I'd like to provide an update on the progress of preparing release
> > > > 1.11.3.
> > > > >
> > > > > *Blockers*
> > > > > We currently have 3 remaining blockers. (3 resolved and 1 emerged
> > > > compared
> > > > > to last week)
> > > > >
> > > > > - [FLINK-19698] Add close() method and onCheckpointComplete() to
> the
> > > > > Source.
> > > > > The issue has been fixed on the master branch. It's currently
> blocked
> > > on
> > > > > the FLIP-27 backportings to backport it to the 1.11 branch.
> > > > >
> > > > > - [FLINK-19717] SourceReaderBase.pollNext may return END_OF_INPUT
> if
> > > > > SplitReader.fetch throws
> > > > > A PR has been opened and reviewed. From the discussions on the PR,
> it
> > > > looks
> > > > > close to mergeable.
> > > > >
> > > > > - [FLINK-19816] Flink restored from a wrong checkpoint (a very old
> > one
> > > > and
> > > > > not the last completed one)
> > > > > This is a newly emerged blocker and Matthias is working on it.
> > > > >
> > > > > *Test Instabilities*
> > > > > We currently have 27 test instabilities[1].
> > > > > AFAIK, none of them are as serious as to block the 1.11.3 release.
> > > > >
> > > > > *FLIP-27 Backprotings*
> > > > >
> > > > > I noticed that there's no jira issues opened on the FLIP-27
> > backporting
> > > > > efforts, which is part of the major efforts planned for the 1.11.3
> > > > release,
> > > > > making it hard to track the progress.
> > > > >
> > > > >
> > > > > @Stephan and @Becket, could you please share the updates on the
> > > > backporting
> > > > > efforts? How is the progress and when are the efforts expected to
> be
> > > > > finished? It would be appreciated and helpful if we can have a jira
> > > > ticket
> > > > > to track the backporing progress publicly.
> > > > >
> > > > >
> > > > > Thank you~
> > > > >
> > > > > Xintong Song
> > > > >
> > > > >
> > > > > [1]
> > > > >
> > > > >
> > > >
> > >
> >
> https://issues.apache.org/jira/browse/FLINK-16908?filter=12348580&jql=project%20%3D%20FLINK%20AND%20resolution%20%3D%20Unresolved%20AND%20affectedVersion%20in%20(1.11.0%2C%201.11.1%2C%201.11.2%2C%201.11.3)%20AND%20labels%20%3D%20test-stability%20ORDER%20BY%20updated%20DESC%2C%20created%20DESC
> > > > >
> > > > > On Wed, Nov 4, 2020 at 3:58 PM Tzu-Li (Gordon) Tai <
> > > tzulitai@apache.org>
> > > > > wrote:
> > > > >
> > > > > > Thanks for the efforts so far with pushing for 1.11.3.
> > > > > >
> > > > > > I'd like to provide a mid-week update on how we're looking with
> all
> > > the
> > > > > > pending blockers and backports:
> > > > > >
> > > > > > *Blockers**:*
> > > > > >
> > > > > > Currently all blockers either have an open PR, or have been
> merged.
> > > > I'll
> > > > > > highlight below the blockers *that still require reviewing
> efforts
> > > for
> > > > > them
> > > > > > to move forward:*
> > > > > >
> > > > > > - [FLINK-19909] Flink application in attach mode could not
> > terminate
> > > > when
> > > > > > the only job is canceled. PR:
> > > > https://github.com/apache/flink/pull/13911
> > > > > > - [FLINK-19717] SourceReaderBase.pollNext may return END_OF_INPUT
> > if
> > > > > > SplitReader.fetch throws. PR:
> > > > https://github.com/apache/flink/pull/13776
> > > > > >
> > > > > > The above PRs currently have no reviews at all yet. They seem to
> > > > already
> > > > > > have designated reviewers.
> > > > > >
> > > > > >
> > > > > > *Backports:*
> > > > > > - FLIP-27 robustness improvement backports: Stephan is currently
> > > > working
> > > > > on
> > > > > > backporting several FLIP-27 changes. There are no PRs yet for the
> > > > > > backported changes.
> > > > > > - DataStreamUtils.collect() refactorings backport: Steven Wu
> > > mentioned
> > > > to
> > > > > > backport this, but AFAIK this isn't assigned to anyone yet.
> > > > > >
> > > > > > *ETAs*:
> > > > > >
> > > > > > I'd like to request ETAs for the remaining backports, to prevent
> a
> > > > creep
> > > > > in
> > > > > > the scope of this bugfix release.
> > > > > >
> > > > > > We already have Flink users that would benefit from fixes that
> have
> > > > been
> > > > > > merged to release-1.11 already, so technically speaking the
> > backports
> > > > > > should be considered "nice-to-have" (to the best of my knowledge
> of
> > > the
> > > > > > changes) and could potentially be moved to a follow-up 1.11.4.
> > > > > > Most notably, the Stateful Functions project is already waiting
> on
> > > > Flink
> > > > > > 1.11.3 to address critical recovery issues (please see the
> StateFun
> > > > 2.2.1
> > > > > > release discussion thread [1]).
> > > > > >
> > > > > > @Stephan Ewen <se...@apache.org> @Becket Qin <
> becket.qin@gmail.com
> > >
> > > > > could
> > > > > > you provide an ETA for the FLIP-27 backports? It would help to
> get
> > a
> > > > > better
> > > > > > estimate to decide how we proceed here.
> > > > > >
> > > > > > Cheers,
> > > > > > Gordon
> > > > > >
> > > > > > [1]
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Releasing-StateFun-hotfix-version-2-2-1-td46239.html
> > > > > >
> > > > > > On Wed, Nov 4, 2020 at 3:16 PM Tzu-Li (Gordon) Tai <
> > > > tzulitai@apache.org>
> > > > > > wrote:
> > > > > >
> > > > > > > > The collect() utils can be picked back, I see no issue with
> > that
> > > > (it
> > > > > is
> > > > > > > isolated utilities).
> > > > > > >
> > > > > > > Just checking on all the requested backports mentioned in this
> > > > thread,
> > > > > > and
> > > > > > > figuring out which ones seem to still be unassigned / open.
> > > > > > >
> > > > > > > Is someone working on backporting
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/flink/commit/09a7a66b7313fea64817fe960a8da1265b428efc
> > > > > > > to release-1.11 at the moment?
> > > > > > >
> > > > > > > On Wed, Nov 4, 2020 at 1:39 AM Steven Wu <stevenz3wu@gmail.com
> >
> > > > wrote:
> > > > > > >
> > > > > > >> @Stephan Ewen <se...@apache.org> yeah, we can do that. don't
> > > worry
> > > > > > about
> > > > > > >> it. your earlier email had the perfect explanation on why file
> > > > source
> > > > > > >> shouldn't be backported.
> > > > > > >>
> > > > > > >> On Tue, Nov 3, 2020 at 3:37 AM Stephan Ewen <sewen@apache.org
> >
> > > > wrote:
> > > > > > >>
> > > > > > >> > @Steven would it be possible to initially copy some of the
> > code
> > > > into
> > > > > > the
> > > > > > >> > iceberg source and later replace it by a dependency on the
> > Flink
> > > > > file
> > > > > > >> > source?
> > > > > > >> >
> > > > > > >> > On Mon, Nov 2, 2020 at 8:33 PM Steven Wu <
> > stevenz3wu@gmail.com>
> > > > > > wrote:
> > > > > > >> >
> > > > > > >> > > Stephan, thanks a lot for explaining the file connector.
> > that
> > > > > makes
> > > > > > >> > sense.
> > > > > > >> > >
> > > > > > >> > > I was asking because we were trying to reuse some of the
> > > > > > >> implementations
> > > > > > >> > in
> > > > > > >> > > the file source for Iceberg source. Flink Iceberg source
> > lives
> > > > in
> > > > > > the
> > > > > > >> > > Iceberg repo, which is not possible to code against the
> > master
> > > > > > branch
> > > > > > >> of
> > > > > > >> > > the Flink code.
> > > > > > >> > >
> > > > > > >> > > On Mon, Nov 2, 2020 at 3:31 AM Stephan Ewen <
> > sewen@apache.org
> > > >
> > > > > > wrote:
> > > > > > >> > >
> > > > > > >> > > > Hi Steven!
> > > > > > >> > > >
> > > > > > >> > > > So far there are no plans to pick back the file system
> > > > connector
> > > > > > >> code.
> > > > > > >> > > This
> > > > > > >> > > > is still evolving and not finalized for 1.12, so I don't
> > > feel
> > > > it
> > > > > > is
> > > > > > >> a
> > > > > > >> > > good
> > > > > > >> > > > candidate to be backported.
> > > > > > >> > > > However, with the base connector changes backported, you
> > > > should
> > > > > be
> > > > > > >> able
> > > > > > >> > > to
> > > > > > >> > > > run the file connector code from master against 1.11.3.
> > > > > > >> > > >
> > > > > > >> > > > The collect() utils can be picked back, I see no issue
> > with
> > > > that
> > > > > > >> (it is
> > > > > > >> > > > isolated utilities).
> > > > > > >> > > >
> > > > > > >> > > > Best,
> > > > > > >> > > > Stephan
> > > > > > >> > > >
> > > > > > >> > > >
> > > > > > >> > > > On Mon, Nov 2, 2020 at 3:02 AM Steven Wu <
> > > > stevenz3wu@gmail.com>
> > > > > > >> wrote:
> > > > > > >> > > >
> > > > > > >> > > > > Basically, it would be great to get the latest code in
> > the
> > > > > > >> > > > > flink-connector-files (FLIP-27).
> > > > > > >> > > > >
> > > > > > >> > > > > On Sat, Oct 31, 2020 at 9:57 AM Steven Wu <
> > > > > stevenz3wu@gmail.com
> > > > > > >
> > > > > > >> > > wrote:
> > > > > > >> > > > >
> > > > > > >> > > > > > Stephan, it will be great if we can also backport
> the
> > > > > > >> > DataStreamUtils
> > > > > > >> > > > > > related commits that help with collecting output
> from
> > > > > > unbounded
> > > > > > >> > > > streams.
> > > > > > >> > > > > > e.g.
> > > > > > >> > > > > >
> > > > > > >> > > > > >
> > > > > > >> > > > > >
> > > > > > >> > > > >
> > > > > > >> > > >
> > > > > > >> > >
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/flink/commit/09a7a66b7313fea64817fe960a8da1265b428efc
> > > > > > >> > > > > >
> > > > > > >> > > > > > I tried to copy and paste the code to unblock
> myself.
> > > but
> > > > it
> > > > > > >> > quickly
> > > > > > >> > > > got
> > > > > > >> > > > > > into the rabbit hole of more and more code.
> > > > > > >> > > > > >
> > > > > > >> > > > > > On Fri, Oct 30, 2020 at 11:02 AM Stephan Ewen <
> > > > > > sewen@apache.org
> > > > > > >> >
> > > > > > >> > > > wrote:
> > > > > > >> > > > > >
> > > > > > >> > > > > >> I have started with backporting the source API
> > changes.
> > > > > Some
> > > > > > >> minor
> > > > > > >> > > > > >> conflicts to solve, will need a bit more to finish
> > > this.
> > > > > > >> > > > > >>
> > > > > > >> > > > > >> On Fri, Oct 30, 2020 at 7:25 AM Tzu-Li (Gordon)
> Tai <
> > > > > > >> > > > > tzulitai@apache.org>
> > > > > > >> > > > > >> wrote:
> > > > > > >> > > > > >>
> > > > > > >> > > > > >> > @Stephan Ewen <se...@apache.org>
> > > > > > >> > > > > >> > Are there already plans or ongoing efforts for
> > > > > backporting
> > > > > > >> the
> > > > > > >> > > list
> > > > > > >> > > > of
> > > > > > >> > > > > >> > FLIP-27 changes that you posted?
> > > > > > >> > > > > >> >
> > > > > > >> > > > > >> > On Thu, Oct 29, 2020 at 7:08 PM Xintong Song <
> > > > > > >> > > tonysong820@gmail.com
> > > > > > >> > > > >
> > > > > > >> > > > > >> > wrote:
> > > > > > >> > > > > >> >
> > > > > > >> > > > > >> >> Hi folks,
> > > > > > >> > > > > >> >>
> > > > > > >> > > > > >> >> Just to provide some updates concerning the
> status
> > > on
> > > > > the
> > > > > > >> > > > > >> >> test instabilities.
> > > > > > >> > > > > >> >>
> > > > > > >> > > > > >> >> Currently, we have 30 unresolved tickets labeled
> > > with
> > > > > > >> `Affects
> > > > > > >> > > > > Version`
> > > > > > >> > > > > >> >> 1.11.x.
> > > > > > >> > > > > >> >>
> > > > > > >> > > > > >> >>
> > > > > > >> > > > > >>
> > > > > > >> > > > >
> > > > > > >> > > >
> > > > > > >> > >
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> https://issues.apache.org/jira/browse/FLINK-19775?filter=12348580&jql=project%20%3D%20FLINK%20AND%20resolution%20%3D%20Unresolved%20AND%20affectedVersion%20in%20(1.11.0%2C%201.11.1%2C%201.11.2%2C%201.11.3)%20AND%20labels%20%3D%20test-stability%20ORDER%20BY%20created%20DESC
> > > > > > >> > > > > >> >>
> > > > > > >> > > > > >> >> Among the 30 tickets, 11 of them are:
> > > > > > >> > > > > >> >> - Have occured in the recent 3 months
> > > > > > >> > > > > >> >> - Not confirmed to be pure testability issues
> > > > > > >> > > > > >> >> - Not confirmed to be rare condition cases
> > > > > > >> > > > > >> >>
> > > > > > >> > > > > >> >> It would be nice if someone familiar with these
> > > > > components
> > > > > > >> can
> > > > > > >> > > > take a
> > > > > > >> > > > > >> look
> > > > > > >> > > > > >> >> into these issues.
> > > > > > >> > > > > >> >>
> > > > > > >> > > > > >> >> -
> > https://issues.apache.org/jira/browse/FLINK-17159
> > > > > (ES6)
> > > > > > >> > > > > >> >> -
> > https://issues.apache.org/jira/browse/FLINK-17912
> > > > > > (Kafka)
> > > > > > >> > > > > >> >> -
> > https://issues.apache.org/jira/browse/FLINK-17949
> > > > > > (Kafka)
> > > > > > >> > > > > >> >> ⁃
> > https://issues.apache.org/jira/browse/FLINK-18444
> > > > > > (Kafka)
> > > > > > >> > > > > >> >> -
> > https://issues.apache.org/jira/browse/FLINK-18634
> > > > > > (Kafka)
> > > > > > >> > > > > >> >> -
> > https://issues.apache.org/jira/browse/FLINK-18648
> > > > > > (Kafka)
> > > > > > >> > > > > >> >> -
> > https://issues.apache.org/jira/browse/FLINK-18807
> > > > > > (Kafka)
> > > > > > >> > > > > >> >> -
> > https://issues.apache.org/jira/browse/FLINK-19369
> > > > > > >> > > > (BlobClientTest)
> > > > > > >> > > > > >> >> -
> > https://issues.apache.org/jira/browse/FLINK-19436
> > > > > > (TPCDS)
> > > > > > >> > > > > >> >> -
> > https://issues.apache.org/jira/browse/FLINK-19690
> > > > > > >> > > > (Format/Parquet)
> > > > > > >> > > > > >> >> -
> > https://issues.apache.org/jira/browse/FLINK-19775
> > > > > > >> > > > > >> >> (SystemProcessingTimeServiceTest)
> > > > > > >> > > > > >> >>
> > > > > > >> > > > > >> >> Thank you~
> > > > > > >> > > > > >> >>
> > > > > > >> > > > > >> >> Xintong Song
> > > > > > >> > > > > >> >>
> > > > > > >> > > > > >> >>
> > > > > > >> > > > > >> >>
> > > > > > >> > > > > >> >> On Thu, Oct 29, 2020 at 10:21 AM Jingsong Li <
> > > > > > >> > > > jingsonglee0@gmail.com
> > > > > > >> > > > > >
> > > > > > >> > > > > >> >> wrote:
> > > > > > >> > > > > >> >>
> > > > > > >> > > > > >> >> > +1 to backport the FLIP-27 adjustments to
> > 1.11.x.
> > > > > > >> > > > > >> >> >
> > > > > > >> > > > > >> >> > If possible, that would be great. Many people
> > are
> > > > > > looking
> > > > > > >> > > forward
> > > > > > >> > > > > to
> > > > > > >> > > > > >> the
> > > > > > >> > > > > >> >> > FLIP-27 interface, but they don't want to take
> > the
> > > > > risk
> > > > > > to
> > > > > > >> > > > upgrade
> > > > > > >> > > > > to
> > > > > > >> > > > > >> >> 1.12
> > > > > > >> > > > > >> >> > (And wait 1.12). After all, 1.11 is a
> relatively
> > > > > stable
> > > > > > >> > > version.
> > > > > > >> > > > > >> >> >
> > > > > > >> > > > > >> >> > Best,
> > > > > > >> > > > > >> >> > Jingsong
> > > > > > >> > > > > >> >> >
> > > > > > >> > > > > >> >> > On Thu, Oct 29, 2020 at 1:24 AM Stephan Ewen <
> > > > > > >> > sewen@apache.org
> > > > > > >> > > >
> > > > > > >> > > > > >> wrote:
> > > > > > >> > > > > >> >> >
> > > > > > >> > > > > >> >> > > Thanks for starting this.
> > > > > > >> > > > > >> >> > >
> > > > > > >> > > > > >> >> > > +1 form my side to backport the FLIP-27
> > > > adjustments
> > > > > to
> > > > > > >> > > 1.11.x.
> > > > > > >> > > > > >> >> > >
> > > > > > >> > > > > >> >> > > There were quite a few changes, and I think
> we
> > > > need
> > > > > to
> > > > > > >> > > > > cherry-pick
> > > > > > >> > > > > >> >> them
> > > > > > >> > > > > >> >> > all
> > > > > > >> > > > > >> >> > > to not get some inconsistent mix of changes
> > and
> > > > many
> > > > > > >> merge
> > > > > > >> > > > > >> conflicts.
> > > > > > >> > > > > >> >> > > I made a list below of what we need to add
> to
> > > > > > >> > "release-1.11".
> > > > > > >> > > > > >> >> > >
> > > > > > >> > > > > >> >> > > *#### Core Source API Changes to backport
> (in
> > > > > REVERSE
> > > > > > >> > order)*
> > > > > > >> > > > > >> >> > >
> > > > > > >> > > > > >> >> > >   (Use: "git log
> > > > > > >> > > > > >> >> > >
> > > > > > >> > > >
> > > > flink-core/src/main/java/org/apache/flink/api/connector/source")
> > > > > > >> > > > > >> >> > >
> > > > > > >> > > > > >> >> > > commit
> > 162c072e9265a7b6dd9d6f5459eb7974091c4c4e
> > > > > > >> > > > > >> >> > > [FLINK-19492][core] Consolidate Source
> Events
> > > > > between
> > > > > > >> > Source
> > > > > > >> > > > API
> > > > > > >> > > > > >> and
> > > > > > >> > > > > >> >> > Split
> > > > > > >> > > > > >> >> > > Reader API
> > > > > > >> > > > > >> >> > >
> > > > > > >> > > > > >> >> > > commit
> > 428fbbad85379f71a9fbf1a1ab3e3dcfa9006436
> > > > > > >> > > > > >> >> > > [FLINK-19205][core] Add access to
> > configuration
> > > > and
> > > > > > >> > hostname
> > > > > > >> > > in
> > > > > > >> > > > > the
> > > > > > >> > > > > >> >> > > SourceReaderContext
> > > > > > >> > > > > >> >> > >
> > > > > > >> > > > > >> >> > > commit
> > 723e1790744ddc78d19c7c978442af1383f38d33
> > > > > > >> > > > > >> >> > > [hotfix][core] Add to Source Enumerator
> > > > convenience
> > > > > > >> methods
> > > > > > >> > > to
> > > > > > >> > > > > >> assign
> > > > > > >> > > > > >> >> > > single split
> > > > > > >> > > > > >> >> > >
> > > > > > >> > > > > >> >> > > *#### Connector Base Changes to Backport (in
> > > > REVERSE
> > > > > > >> > order)*
> > > > > > >> > > > > >> >> > >
> > > > > > >> > > > > >> >> > >   (Use: "git log
> > > > > > flink-connectors/flink-connector-base")
> > > > > > >> > > > > >> >> > >
> > > > > > >> > > > > >> >> > > commit
> > 401f56fe9d6b0271260edf9787cdcbfe4d03874d
> > > > > > >> > > > > >> >> > > [FLINK-19427][FLINK-19489][tests] Fix test
> > > > > conditions
> > > > > > >> for
> > > > > > >> > > > > >> >> > >
> > > > > > 'SplitFetcherTest.testNotifiesWhenGoingIdleConcurrent()'
> > > > > > >> > > > > >> >> > >
> > > > > > >> > > > > >> >> > > commit
> > 68c5c2ff779d82a1ff81ffaf60d8a1b283797db1
> > > > > > >> > > > > >> >> > > [FLINK-19448][connector base] Explicitly
> check
> > > for
> > > > > > >> > > un-expected
> > > > > > >> > > > > >> >> condition
> > > > > > >> > > > > >> >> > > that would leave an inconsistent state
> > > > > > >> > > > > >> >> > >
> > > > > > >> > > > > >> >> > > commit
> > 162c072e9265a7b6dd9d6f5459eb7974091c4c4e
> > > > > > >> > > > > >> >> > > [FLINK-19492][core] Consolidate Source
> Events
> > > > > between
> > > > > > >> > Source
> > > > > > >> > > > API
> > > > > > >> > > > > >> and
> > > > > > >> > > > > >> >> > Split
> > > > > > >> > > > > >> >> > > Reader API
> > > > > > >> > > > > >> >> > >
> > > > > > >> > > > > >> >> > > commit
> > c1ca7a4c7c21ec8868c14cdddd43c559625b794c
> > > > > > >> > > > > >> >> > > [refactor][tests] Move some source test
> utils
> > > from
> > > > > > >> > > > > >> >> flink-connector-base
> > > > > > >> > > > > >> >> > to
> > > > > > >> > > > > >> >> > > flink-core
> > > > > > >> > > > > >> >> > >
> > > > > > >> > > > > >> >> > > commit
> > ee5c4c211c35c70d28252363bbc8400453609977
> > > > > > >> > > > > >> >> > > [FLINK-19251][connectors] Avoid confusing
> > queue
> > > > > > >> handling in
> > > > > > >> > > > > >> >> > > "SplitReader.handleSplitsChanges()"
> > > > > > >> > > > > >> >> > >
> > > > > > >> > > > > >> >> > > commit
> > 5abef56b2bf85bcac786f6b16b6899b6cced7176
> > > > > > >> > > > > >> >> > > [FLINK-19250][connectors] Fix error
> > propagation
> > > in
> > > > > > >> > connector
> > > > > > >> > > > base
> > > > > > >> > > > > >> >> > > (SplitFetcherManager).
> > > > > > >> > > > > >> >> > >
> > > > > > >> > > > > >> >> > > commit
> > 8fcca837c55a9216595ee4c03038b52747098dbb
> > > > > > >> > > > > >> >> > > [hotfix][connectors] Improve JavaDocs for
> > > > > > >> > > > > >> SingleThreadFetcherManager
> > > > > > >> > > > > >> >> > >
> > > > > > >> > > > > >> >> > > commit
> > 4700bb5dde3303cbe98882f6beb7379425717b01
> > > > > > >> > > > > >> >> > > [FLINK-19225][connectors] Various small
> > > > improvements
> > > > > > to
> > > > > > >> > > > > >> >> SourceReaderBase
> > > > > > >> > > > > >> >> > > (part 2)
> > > > > > >> > > > > >> >> > >
> > > > > > >> > > > > >> >> > > commit
> > 12261c6b7ed6478a9b9f6a69cb58246b83cab9b7
> > > > > > >> > > > > >> >> > > [FLINK-17393][connectors] (follow-up) Wakeup
> > the
> > > > > > >> > > SplitFetchers
> > > > > > >> > > > > more
> > > > > > >> > > > > >> >> > > elegantly.
> > > > > > >> > > > > >> >> > >
> > > > > > >> > > > > >> >> > > commit
> > c60aaff0249bfd6b5871b7f82e03efc487a54d6b
> > > > > > >> > > > > >> >> > > [hotfix][tests] Extend test coverage for
> > > > > > >> > > > > >> FutureCompletingBlockingQueue
> > > > > > >> > > > > >> >> > >
> > > > > > >> > > > > >> >> > > commit
> > cef8a587d7fd2fe64cc644da5ed095d82e46f631
> > > > > > >> > > > > >> >> > > [FLINK-19245][connectors] Set default
> capacity
> > > for
> > > > > > >> > > > > >> >> > > FutureCompletingBlockingQueue.
> > > > > > >> > > > > >> >> > >
> > > > > > >> > > > > >> >> > > commit
> > 4ea95782b4c6a2538153d4d16ad3f4839c7de0fb
> > > > > > >> > > > > >> >> > > [FLINK-19223][connectors] Simplify
> > Availability
> > > > > Future
> > > > > > >> > Model
> > > > > > >> > > in
> > > > > > >> > > > > >> Base
> > > > > > >> > > > > >> >> > > Connector
> > > > > > >> > > > > >> >> > >
> > > > > > >> > > > > >> >> > > commit
> > 511857049ba30c8ff0ee56da551fa4a479dc583e
> > > > > > >> > > > > >> >> > > [FLINK-18128][connectors] Ensure idle split
> > > > fetchers
> > > > > > >> lead
> > > > > > >> > to
> > > > > > >> > > > > >> >> availability
> > > > > > >> > > > > >> >> > > notifications.
> > > > > > >> > > > > >> >> > >
> > > > > > >> > > > > >> >> > > commit
> > a8206467af0830dcb89623ea068b5ca3b3450c92
> > > > > > >> > > > > >> >> > > [refactor][core] Eagerly initialize the
> > > FetchTask
> > > > to
> > > > > > >> > support
> > > > > > >> > > > > proper
> > > > > > >> > > > > >> >> unit
> > > > > > >> > > > > >> >> > > testing
> > > > > > >> > > > > >> >> > >
> > > > > > >> > > > > >> >> > > commit
> > 3b2f54bcb437f98e6137c904045cc51072b5c06b
> > > > > > >> > > > > >> >> > > [hotfix][tests] Move constants in
> > > SplitFetcherTest
> > > > > > >> relevant
> > > > > > >> > > to
> > > > > > >> > > > > only
> > > > > > >> > > > > >> >> one
> > > > > > >> > > > > >> >> > > test into test method
> > > > > > >> > > > > >> >> > >
> > > > > > >> > > > > >> >> > > commit
> > d7625760a75a508bf05bcddc380bb4d62ee1743e
> > > > > > >> > > > > >> >> > > [FLINK-19225][connectors] Various small
> > > > improvements
> > > > > > to
> > > > > > >> > > > > >> >> SourceReaderBase
> > > > > > >> > > > > >> >> > >
> > > > > > >> > > > > >> >> > > commit
> > a5b0d3297748c1be47ad579a88f24df2255a8df1
> > > > > > >> > > > > >> >> > > [FLINK-17393][connectors] Wakeup the
> > > SplitFetchers
> > > > > > more
> > > > > > >> > > > > elegantly.
> > > > > > >> > > > > >> >> > >
> > > > > > >> > > > > >> >> > > commit
> > f42a3ebc3e81a034b7221a803c153636fef34903
> > > > > > >> > > > > >> >> > > [FLINK-18680][connectors] Make connector
> base
> > > > > > >> > > > RecordsWithSplitIds
> > > > > > >> > > > > >> more
> > > > > > >> > > > > >> >> > > lightweight.
> > > > > > >> > > > > >> >> > >
> > > > > > >> > > > > >> >> > > commit
> > e3d273de822b085183d09b275a445879ff94b350
> > > > > > >> > > > > >> >> > > [FLINK-19162][connectors] Add 'recycle()' to
> > the
> > > > > > >> > > > > >> RecordsWithSplitIds
> > > > > > >> > > > > >> >> to
> > > > > > >> > > > > >> >> > > support reuse of heavy objects.
> > > > > > >> > > > > >> >> > >
> > > > > > >> > > > > >> >> > > commit
> > 8ebc464c2520453a70001cd712abc8dee6ee89e0
> > > > > > >> > > > > >> >> > > [hotfix][testing] Add a set of
> parameterizable
> > > > > testing
> > > > > > >> > mocks
> > > > > > >> > > > for
> > > > > > >> > > > > >> the
> > > > > > >> > > > > >> >> > Split
> > > > > > >> > > > > >> >> > > Reader API
> > > > > > >> > > > > >> >> > >
> > > > > > >> > > > > >> >> > > commit
> > 930a07438be1185388d7150640f294dfe2a5d378
> > > > > > >> > > > > >> >> > > [hotfix][connectors] Add RequestSplitEvent
> to
> > > > > > >> > > > > >> 'flink-connector-base'
> > > > > > >> > > > > >> >> > >
> > > > > > >> > > > > >> >> > > commit
> > e945ce8a933bc378844782f784ca473c767ca159
> > > > > > >> > > > > >> >> > > [refactor][connectors] Backport of the
> > > > > connector-base
> > > > > > >> > > exception
> > > > > > >> > > > > >> >> handling
> > > > > > >> > > > > >> >> > > from the Kafka Connector Pull Request
> > > > > > >> > > > > >> >> > >
> > > > > > >> > > > > >> >> > >
> > > > > > >> > > > > >> >> > > On Wed, Oct 28, 2020 at 5:49 PM Steven Wu <
> > > > > > >> > > > stevenz3wu@gmail.com>
> > > > > > >> > > > > >> >> wrote:
> > > > > > >> > > > > >> >> > >
> > > > > > >> > > > > >> >> > > > I would love to see this FLIP-27 source
> > > > interface
> > > > > > >> > > improvement
> > > > > > >> > > > > [1]
> > > > > > >> > > > > >> >> made
> > > > > > >> > > > > >> >> > to
> > > > > > >> > > > > >> >> > > > 1.11.3.
> > > > > > >> > > > > >> >> > > >
> > > > > > >> > > > > >> >> > > > [1]
> > > > > > https://issues.apache.org/jira/browse/FLINK-19698
> > > > > > >> > > > > >> >> > > >
> > > > > > >> > > > > >> >> > > > On Wed, Oct 28, 2020 at 12:32 AM Tzu-Li
> > > (Gordon)
> > > > > > Tai <
> > > > > > >> > > > > >> >> > > tzulitai@apache.org>
> > > > > > >> > > > > >> >> > > > wrote:
> > > > > > >> > > > > >> >> > > >
> > > > > > >> > > > > >> >> > > > > Thanks for the replies so far!
> > > > > > >> > > > > >> >> > > > >
> > > > > > >> > > > > >> >> > > > > Just to provide a brief update on the
> > status
> > > > of
> > > > > > >> > blockers
> > > > > > >> > > > for
> > > > > > >> > > > > >> >> 1.11.3
> > > > > > >> > > > > >> >> > so
> > > > > > >> > > > > >> >> > > > far:
> > > > > > >> > > > > >> >> > > > >
> > > > > > >> > > > > >> >> > > > >
> > > > > > >> > > > > >> >> > > > > *PR opened, pending reviewer*-
> > [FLINK-19717]
> > > > > > >> > > > > >> >> > SourceReaderBase.pollNext
> > > > > > >> > > > > >> >> > > > may
> > > > > > >> > > > > >> >> > > > > return END_OF_INPUT if SplitReader.fetch
> > > > throws
> > > > > (
> > > > > > >> > > > > >> >> > > > >
> > https://github.com/apache/flink/pull/13776)
> > > > > > >> > > > > >> >> > > > >
> > > > > > >> > > > > >> >> > > > > *PR opened, reviewed + close to being
> > > merged*
> > > > > > >> > > > > >> >> > > > > - [FLINK-19741] Timer service should
> skip
> > > > > > restoring
> > > > > > >> > from
> > > > > > >> > > > raw
> > > > > > >> > > > > >> keyed
> > > > > > >> > > > > >> >> > > stream
> > > > > > >> > > > > >> >> > > > > if it isn't the writer (
> > > > > > >> > > > > >> >> https://github.com/apache/flink/pull/13761)
> > > > > > >> > > > > >> >> > > > > - [FLINK-19748] Raw keyed stream key
> group
> > > > > > iterator
> > > > > > >> > > should
> > > > > > >> > > > be
> > > > > > >> > > > > >> >> > skipping
> > > > > > >> > > > > >> >> > > > > unwritten key groups (
> > > > > > >> > > > > >> https://github.com/apache/flink/pull/13772)
> > > > > > >> > > > > >> >> > > > >
> > > > > > >> > > > > >> >> > > > > *Merged*
> > > > > > >> > > > > >> >> > > > > - [FLINK-19154] Application mode deletes
> > HA
> > > > data
> > > > > > in
> > > > > > >> > case
> > > > > > >> > > of
> > > > > > >> > > > > >> >> suspended
> > > > > > >> > > > > >> >> > > > > ZooKeeper connection
> > > > > > >> > > > > >> >> > > > > - [FLINK-19569] Upgrade ICU4J to 67.1+
> > > > > > >> > > > > >> >> > > > >
> > > > > > >> > > > > >> >> > > > > Right now as it seems, progress is
> mainly
> > > > > blocked
> > > > > > >> on a
> > > > > > >> > > > > reviewer
> > > > > > >> > > > > >> >> for
> > > > > > >> > > > > >> >> > > > > FLINK-19717.
> > > > > > >> > > > > >> >> > > > > Meanwhile, Xintong is keeping an eye on
> > test
> > > > > > >> > > instabilities
> > > > > > >> > > > > [1]
> > > > > > >> > > > > >> to
> > > > > > >> > > > > >> >> see
> > > > > > >> > > > > >> >> > > if
> > > > > > >> > > > > >> >> > > > > there are any fixes that should be
> applied
> > > to
> > > > > > >> > > > `release-1.11`.
> > > > > > >> > > > > >> >> > > > >
> > > > > > >> > > > > >> >> > > > > This is also a reminder, that if there
> are
> > > > other
> > > > > > >> > blockers
> > > > > > >> > > > > that
> > > > > > >> > > > > >> we
> > > > > > >> > > > > >> >> > need
> > > > > > >> > > > > >> >> > > to
> > > > > > >> > > > > >> >> > > > > be aware of, or a need to re-establish
> > > > estimated
> > > > > > >> time
> > > > > > >> > for
> > > > > > >> > > > > >> getting
> > > > > > >> > > > > >> >> > fixes
> > > > > > >> > > > > >> >> > > > in
> > > > > > >> > > > > >> >> > > > > and delay the RC for 1.11.3, please do
> let
> > > us
> > > > > > know!
> > > > > > >> > > > > >> >> > > > >
> > > > > > >> > > > > >> >> > > > > Cheers,
> > > > > > >> > > > > >> >> > > > > Gordon
> > > > > > >> > > > > >> >> > > > >
> > > > > > >> > > > > >> >> > > > > [1]
> > > > > > >> > > > > >> >> > > > >
> > > > > > >> > > > > >> >> > > > >
> > > > > > >> > > > > >> >> > > >
> > > > > > >> > > > > >> >> > >
> > > > > > >> > > > > >> >> >
> > > > > > >> > > > > >> >>
> > > > > > >> > > > > >>
> > > > > > >> > > > >
> > > > > > >> > > >
> > > > > > >> > >
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> https://issues.apache.org/jira/issues/?jql=project%20%3D%20FLINK%20AND%20status%20in%20(Open%2C%20%22In%20Progress%22%2C%20Reopened)%20AND%20affectedVersion%20in%20(1.11.0%2C%201.11.1%2C%201.11.2)%20AND%20labels%20%3D%20test-stability
> > > > > > >> > > > > >> >> > > > >
> > > > > > >> > > > > >> >> > > > > On Mon, Oct 26, 2020 at 9:43 PM Kostas
> > > > Kloudas <
> > > > > > >> > > > > >> >> kkloudas@apache.org>
> > > > > > >> > > > > >> >> > > > > wrote:
> > > > > > >> > > > > >> >> > > > >
> > > > > > >> > > > > >> >> > > > > > +1 for releasing Flink 1.11.3 as it
> > > > contains a
> > > > > > >> number
> > > > > > >> > > of
> > > > > > >> > > > > >> >> important
> > > > > > >> > > > > >> >> > > > > > fixes and thanks Gordon and Xintong
> for
> > > > > > >> volunteering.
> > > > > > >> > > > > >> >> > > > > >
> > > > > > >> > > > > >> >> > > > > > Cheers,
> > > > > > >> > > > > >> >> > > > > > Kostas
> > > > > > >> > > > > >> >> > > > > >
> > > > > > >> > > > > >> >> > > > > > On Mon, Oct 26, 2020 at 4:37 AM Yu Li
> <
> > > > > > >> > > carp84@gmail.com>
> > > > > > >> > > > > >> wrote:
> > > > > > >> > > > > >> >> > > > > > >
> > > > > > >> > > > > >> >> > > > > > > +1 for releasing Flink 1.11.3, and
> > > thanks
> > > > > > Gordon
> > > > > > >> > and
> > > > > > >> > > > > >> Xintong
> > > > > > >> > > > > >> >> for
> > > > > > >> > > > > >> >> > > > > > > volunteering as our release
> managers.
> > > > > > >> > > > > >> >> > > > > > >
> > > > > > >> > > > > >> >> > > > > > > Best Regards,
> > > > > > >> > > > > >> >> > > > > > > Yu
> > > > > > >> > > > > >> >> > > > > > >
> > > > > > >> > > > > >> >> > > > > > >
> > > > > > >> > > > > >> >> > > > > > > On Mon, 26 Oct 2020 at 09:45,
> Xintong
> > > > Song <
> > > > > > >> > > > > >> >> > tonysong820@gmail.com>
> > > > > > >> > > > > >> >> > > > > > wrote:
> > > > > > >> > > > > >> >> > > > > > >
> > > > > > >> > > > > >> >> > > > > > > > Thanks Gordan for starting this
> > > > > discussion.
> > > > > > >> > > > > >> >> > > > > > > > My pleasure to help with the
> release
> > > > > > process.
> > > > > > >> > > > > >> >> > > > > > > >
> > > > > > >> > > > > >> >> > > > > > > > Thank you~
> > > > > > >> > > > > >> >> > > > > > > >
> > > > > > >> > > > > >> >> > > > > > > > Xintong Song
> > > > > > >> > > > > >> >> > > > > > > >
> > > > > > >> > > > > >> >> > > > > > > >
> > > > > > >> > > > > >> >> > > > > > > >
> > > > > > >> > > > > >> >> > > > > > > > On Fri, Oct 23, 2020 at 11:29 PM
> > Till
> > > > > > >> Rohrmann <
> > > > > > >> > > > > >> >> > > > trohrmann@apache.org
> > > > > > >> > > > > >> >> > > > > >
> > > > > > >> > > > > >> >> > > > > > > > wrote:
> > > > > > >> > > > > >> >> > > > > > > >
> > > > > > >> > > > > >> >> > > > > > > > > Thanks for starting this
> > discussion
> > > > > > Gordon.
> > > > > > >> > There
> > > > > > >> > > > are
> > > > > > >> > > > > >> over
> > > > > > >> > > > > >> >> > 100
> > > > > > >> > > > > >> >> > > > > issues
> > > > > > >> > > > > >> >> > > > > > > > > which are fixed for 1.11.3.
> Hence
> > +1
> > > > > for a
> > > > > > >> > > soonish
> > > > > > >> > > > > >> 1.11.3
> > > > > > >> > > > > >> >> > > > release.
> > > > > > >> > > > > >> >> > > > > > Thanks
> > > > > > >> > > > > >> >> > > > > > > > > for volunteering as our release
> > > > managers
> > > > > > >> Gordon
> > > > > > >> > > and
> > > > > > >> > > > > >> >> Xintong!
> > > > > > >> > > > > >> >> > > > > > > > >
> > > > > > >> > > > > >> >> > > > > > > > > Cheers,
> > > > > > >> > > > > >> >> > > > > > > > > Till
> > > > > > >> > > > > >> >> > > > > > > > >
> > > > > > >> > > > > >> >> > > > > > > > > On Fri, Oct 23, 2020 at 5:02 PM
> > > Tzu-Li
> > > > > > >> (Gordon)
> > > > > > >> > > > Tai <
> > > > > > >> > > > > >> >> > > > > > tzulitai@apache.org
> > > > > > >> > > > > >> >> > > > > > > > >
> > > > > > >> > > > > >> >> > > > > > > > > wrote:
> > > > > > >> > > > > >> >> > > > > > > > >
> > > > > > >> > > > > >> >> > > > > > > > >> Hi,
> > > > > > >> > > > > >> >> > > > > > > > >>
> > > > > > >> > > > > >> >> > > > > > > > >> Xintong and I would like to
> > start a
> > > > > > >> discussion
> > > > > > >> > > for
> > > > > > >> > > > > >> >> releasing
> > > > > > >> > > > > >> >> > > > Flink
> > > > > > >> > > > > >> >> > > > > > > > 1.11.3
> > > > > > >> > > > > >> >> > > > > > > > >> soon.
> > > > > > >> > > > > >> >> > > > > > > > >>
> > > > > > >> > > > > >> >> > > > > > > > >> It seems like we already have a
> > few
> > > > > > >> pressing
> > > > > > >> > > > issues
> > > > > > >> > > > > >> that
> > > > > > >> > > > > >> >> > needs
> > > > > > >> > > > > >> >> > > > to
> > > > > > >> > > > > >> >> > > > > be
> > > > > > >> > > > > >> >> > > > > > > > >> included in a new hotfix
> release:
> > > > > > >> > > > > >> >> > > > > > > > >>
> > > > > > >> > > > > >> >> > > > > > > > >>    - Heap-based timers’ restore
> > > > > behaviour
> > > > > > >> is
> > > > > > >> > > > > causing a
> > > > > > >> > > > > >> >> > > critical
> > > > > > >> > > > > >> >> > > > > > recovery
> > > > > > >> > > > > >> >> > > > > > > > >>    issue for StateFun [1] [2]
> > [3].
> > > > > > >> > > > > >> >> > > > > > > > >>    - There are several
> robustness
> > > > > issues
> > > > > > >> for
> > > > > > >> > the
> > > > > > >> > > > > >> FLIP-27
> > > > > > >> > > > > >> >> new
> > > > > > >> > > > > >> >> > > > > source
> > > > > > >> > > > > >> >> > > > > > API,
> > > > > > >> > > > > >> >> > > > > > > > >>    such as [4]. We already have
> > > some
> > > > > > users
> > > > > > >> > using
> > > > > > >> > > > the
> > > > > > >> > > > > >> >> FLIP-27
> > > > > > >> > > > > >> >> > > API
> > > > > > >> > > > > >> >> > > > > > with
> > > > > > >> > > > > >> >> > > > > > > > >> 1.11.x,
> > > > > > >> > > > > >> >> > > > > > > > >>    so it would be important to
> > get
> > > > > those
> > > > > > >> fixes
> > > > > > >> > > in
> > > > > > >> > > > > for
> > > > > > >> > > > > >> >> 1.11.x
> > > > > > >> > > > > >> >> > > as
> > > > > > >> > > > > >> >> > > > > > well.
> > > > > > >> > > > > >> >> > > > > > > > >>
> > > > > > >> > > > > >> >> > > > > > > > >> Apart from the issues that are
> > > > already
> > > > > > >> marked
> > > > > > >> > as
> > > > > > >> > > > > >> blocker
> > > > > > >> > > > > >> >> for
> > > > > > >> > > > > >> >> > > > > 1.11.3
> > > > > > >> > > > > >> >> > > > > > in
> > > > > > >> > > > > >> >> > > > > > > > our
> > > > > > >> > > > > >> >> > > > > > > > >> JIRA [5], please let us know in
> > > this
> > > > > > >> thread if
> > > > > > >> > > > there
> > > > > > >> > > > > >> is
> > > > > > >> > > > > >> >> > > already
> > > > > > >> > > > > >> >> > > > > > ongoing
> > > > > > >> > > > > >> >> > > > > > > > >> work for other important fixes
> > that
> > > > we
> > > > > > >> should
> > > > > > >> > > try
> > > > > > >> > > > to
> > > > > > >> > > > > >> >> > include.
> > > > > > >> > > > > >> >> > > > > > > > >>
> > > > > > >> > > > > >> >> > > > > > > > >> Xintong and I would like to
> > > volunteer
> > > > > for
> > > > > > >> > > managing
> > > > > > >> > > > > >> this
> > > > > > >> > > > > >> >> > > release,
> > > > > > >> > > > > >> >> > > > > and
> > > > > > >> > > > > >> >> > > > > > > > will
> > > > > > >> > > > > >> >> > > > > > > > >> try to communicate the priority
> > of
> > > > > > pending
> > > > > > >> > > > blockers
> > > > > > >> > > > > >> over
> > > > > > >> > > > > >> >> the
> > > > > > >> > > > > >> >> > > > next
> > > > > > >> > > > > >> >> > > > > > few
> > > > > > >> > > > > >> >> > > > > > > > >> days.
> > > > > > >> > > > > >> >> > > > > > > > >> Since the aforementioned issues
> > are
> > > > > quite
> > > > > > >> > > > critical,
> > > > > > >> > > > > >> we’d
> > > > > > >> > > > > >> >> > like
> > > > > > >> > > > > >> >> > > to
> > > > > > >> > > > > >> >> > > > > aim
> > > > > > >> > > > > >> >> > > > > > > > >> for a *feature
> > > > > > >> > > > > >> >> > > > > > > > >> freeze by the end of next week
> > > (Oct.
> > > > > > 30th)*
> > > > > > >> > and
> > > > > > >> > > > > start
> > > > > > >> > > > > >> the
> > > > > > >> > > > > >> >> > > > release
> > > > > > >> > > > > >> >> > > > > > voting
> > > > > > >> > > > > >> >> > > > > > > > >> process the week after.
> > > > > > >> > > > > >> >> > > > > > > > >> If that is too short of a
> notice
> > > and
> > > > > you
> > > > > > >> might
> > > > > > >> > > > need
> > > > > > >> > > > > >> more
> > > > > > >> > > > > >> >> > time,
> > > > > > >> > > > > >> >> > > > > > please
> > > > > > >> > > > > >> >> > > > > > > > let
> > > > > > >> > > > > >> >> > > > > > > > >> us know!
> > > > > > >> > > > > >> >> > > > > > > > >>
> > > > > > >> > > > > >> >> > > > > > > > >> Cheers,
> > > > > > >> > > > > >> >> > > > > > > > >> Gordon
> > > > > > >> > > > > >> >> > > > > > > > >>
> > > > > > >> > > > > >> >> > > > > > > > >> [1]
> > > > > > >> > > > > https://issues.apache.org/jira/browse/FLINK-19692
> > > > > > >> > > > > >> >> > > > > > > > >> [2]
> > > > > > >> > > > > https://issues.apache.org/jira/browse/FLINK-19741
> > > > > > >> > > > > >> >> > > > > > > > >> [3]
> > > > > > >> > > > > https://issues.apache.org/jira/browse/FLINK-19748
> > > > > > >> > > > > >> >> > > > > > > > >> [4]
> > > > > > >> > > > > https://issues.apache.org/jira/browse/FLINK-19717
> > > > > > >> > > > > >> >> > > > > > > > >> [5]
> > > > > > >> > > > > >> >> > > > > > > > >>
> > > > > > >> > > > > >> >> > > > > > > > >>
> > > > > > >> > > > > >> >> > > > > > > >
> > > > > > >> > > > > >> >> > > > > >
> > > > > > >> > > > > >> >> > > > >
> > > > > > >> > > > > >> >> > > >
> > > > > > >> > > > > >> >> > >
> > > > > > >> > > > > >> >> >
> > > > > > >> > > > > >> >>
> > > > > > >> > > > > >>
> > > > > > >> > > > >
> > > > > > >> > > >
> > > > > > >> > >
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> https://issues.apache.org/jira/issues/?jql=project%20%3D%20FLINK%20AND%20priority%20%3D%20Blocker%20AND%20fixVersion%20%3D%201.11.3
> > > > > > >> > > > > >> >> > > > > > > > >>
> > > > > > >> > > > > >> >> > > > > > > > >
> > > > > > >> > > > > >> >> > > > > > > >
> > > > > > >> > > > > >> >> > > > > >
> > > > > > >> > > > > >> >> > > > >
> > > > > > >> > > > > >> >> > > >
> > > > > > >> > > > > >> >> > >
> > > > > > >> > > > > >> >> >
> > > > > > >> > > > > >> >> >
> > > > > > >> > > > > >> >> > --
> > > > > > >> > > > > >> >> > Best, Jingsong Lee
> > > > > > >> > > > > >> >> >
> > > > > > >> > > > > >> >>
> > > > > > >> > > > > >> >
> > > > > > >> > > > > >>
> > > > > > >> > > > > >
> > > > > > >> > > > >
> > > > > > >> > > >
> > > > > > >> > >
> > > > > > >> >
> > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] Releasing Apache Flink 1.11.3

Posted by Becket Qin <be...@gmail.com>.
Hi Xintong,

Thanks for driving the release. Just want to sync up on the FLIP-27
backporting. Stephan and I are still trying to backport a bunch of patches
of Source to 1.11.3. Including:

[FLINK-19698][connector/common] Add a close() method to the SplitReader.
[FLINK-19717] SourceReaderBase.pollNext may return END_OF_INPUT if
SplitReader.fetch throws
[FLINK-19535] [connector/common] Avoid failing a job multiple times in
SourceCoordinator.
[FLINK-19265] [FLINK-20049][core] Source API final adjustments.

and a few more fixes.

We are currently trying to fix them in 1.12 first so it might take a little
longer to backport them to 1.11.3. I think it will probably take us a few
more days to finish the backport. So that would roughly be the end of this
week.

Thanks,

Jiangjie (Becket) Qin




On Mon, Nov 9, 2020 at 9:57 PM Till Rohrmann <tr...@apache.org> wrote:

> Yes, I've downgraded FLINK-19816 to critical.
>
> Cheers,
> Till
>
> On Mon, Nov 9, 2020 at 10:19 AM Xintong Song <to...@gmail.com>
> wrote:
>
> > Thanks for the notice, Till.
> >
> > I just checked and found FLINK-20033 is already fixed. Shall we also
> > downgrade FLINK-19816 to `Critical`?
> >
> > Thank you~
> >
> > Xintong Song
> >
> >
> >
> > On Mon, Nov 9, 2020 at 4:42 PM Till Rohrmann <tr...@apache.org>
> wrote:
> >
> > > I would like to bring one more critical issue to your attention which
> is
> > > FLINK-20033 [1]. I believe that this issue is actually causing what has
> > > been reported in FLINK-19816 [2]. I hope to have it fixed by the end of
> > > today. Once FLINK-20033 is fixed, I think that we don't have to block
> the
> > > release on FLINK-19816.
> > >
> > > [1] https://issues.apache.org/jira/browse/FLINK-20033
> > > [2] https://issues.apache.org/jira/browse/FLINK-19816
> > >
> > > Cheers,
> > > Till
> > >
> > > On Mon, Nov 9, 2020 at 4:05 AM Xintong Song <to...@gmail.com>
> > wrote:
> > >
> > > > Hi devs,
> > > >
> > > > I'd like to provide an update on the progress of preparing release
> > > 1.11.3.
> > > >
> > > > *Blockers*
> > > > We currently have 3 remaining blockers. (3 resolved and 1 emerged
> > > compared
> > > > to last week)
> > > >
> > > > - [FLINK-19698] Add close() method and onCheckpointComplete() to the
> > > > Source.
> > > > The issue has been fixed on the master branch. It's currently blocked
> > on
> > > > the FLIP-27 backportings to backport it to the 1.11 branch.
> > > >
> > > > - [FLINK-19717] SourceReaderBase.pollNext may return END_OF_INPUT if
> > > > SplitReader.fetch throws
> > > > A PR has been opened and reviewed. From the discussions on the PR, it
> > > looks
> > > > close to mergeable.
> > > >
> > > > - [FLINK-19816] Flink restored from a wrong checkpoint (a very old
> one
> > > and
> > > > not the last completed one)
> > > > This is a newly emerged blocker and Matthias is working on it.
> > > >
> > > > *Test Instabilities*
> > > > We currently have 27 test instabilities[1].
> > > > AFAIK, none of them are as serious as to block the 1.11.3 release.
> > > >
> > > > *FLIP-27 Backprotings*
> > > >
> > > > I noticed that there's no jira issues opened on the FLIP-27
> backporting
> > > > efforts, which is part of the major efforts planned for the 1.11.3
> > > release,
> > > > making it hard to track the progress.
> > > >
> > > >
> > > > @Stephan and @Becket, could you please share the updates on the
> > > backporting
> > > > efforts? How is the progress and when are the efforts expected to be
> > > > finished? It would be appreciated and helpful if we can have a jira
> > > ticket
> > > > to track the backporing progress publicly.
> > > >
> > > >
> > > > Thank you~
> > > >
> > > > Xintong Song
> > > >
> > > >
> > > > [1]
> > > >
> > > >
> > >
> >
> https://issues.apache.org/jira/browse/FLINK-16908?filter=12348580&jql=project%20%3D%20FLINK%20AND%20resolution%20%3D%20Unresolved%20AND%20affectedVersion%20in%20(1.11.0%2C%201.11.1%2C%201.11.2%2C%201.11.3)%20AND%20labels%20%3D%20test-stability%20ORDER%20BY%20updated%20DESC%2C%20created%20DESC
> > > >
> > > > On Wed, Nov 4, 2020 at 3:58 PM Tzu-Li (Gordon) Tai <
> > tzulitai@apache.org>
> > > > wrote:
> > > >
> > > > > Thanks for the efforts so far with pushing for 1.11.3.
> > > > >
> > > > > I'd like to provide a mid-week update on how we're looking with all
> > the
> > > > > pending blockers and backports:
> > > > >
> > > > > *Blockers**:*
> > > > >
> > > > > Currently all blockers either have an open PR, or have been merged.
> > > I'll
> > > > > highlight below the blockers *that still require reviewing efforts
> > for
> > > > them
> > > > > to move forward:*
> > > > >
> > > > > - [FLINK-19909] Flink application in attach mode could not
> terminate
> > > when
> > > > > the only job is canceled. PR:
> > > https://github.com/apache/flink/pull/13911
> > > > > - [FLINK-19717] SourceReaderBase.pollNext may return END_OF_INPUT
> if
> > > > > SplitReader.fetch throws. PR:
> > > https://github.com/apache/flink/pull/13776
> > > > >
> > > > > The above PRs currently have no reviews at all yet. They seem to
> > > already
> > > > > have designated reviewers.
> > > > >
> > > > >
> > > > > *Backports:*
> > > > > - FLIP-27 robustness improvement backports: Stephan is currently
> > > working
> > > > on
> > > > > backporting several FLIP-27 changes. There are no PRs yet for the
> > > > > backported changes.
> > > > > - DataStreamUtils.collect() refactorings backport: Steven Wu
> > mentioned
> > > to
> > > > > backport this, but AFAIK this isn't assigned to anyone yet.
> > > > >
> > > > > *ETAs*:
> > > > >
> > > > > I'd like to request ETAs for the remaining backports, to prevent a
> > > creep
> > > > in
> > > > > the scope of this bugfix release.
> > > > >
> > > > > We already have Flink users that would benefit from fixes that have
> > > been
> > > > > merged to release-1.11 already, so technically speaking the
> backports
> > > > > should be considered "nice-to-have" (to the best of my knowledge of
> > the
> > > > > changes) and could potentially be moved to a follow-up 1.11.4.
> > > > > Most notably, the Stateful Functions project is already waiting on
> > > Flink
> > > > > 1.11.3 to address critical recovery issues (please see the StateFun
> > > 2.2.1
> > > > > release discussion thread [1]).
> > > > >
> > > > > @Stephan Ewen <se...@apache.org> @Becket Qin <becket.qin@gmail.com
> >
> > > > could
> > > > > you provide an ETA for the FLIP-27 backports? It would help to get
> a
> > > > better
> > > > > estimate to decide how we proceed here.
> > > > >
> > > > > Cheers,
> > > > > Gordon
> > > > >
> > > > > [1]
> > > > >
> > > > >
> > > >
> > >
> >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Releasing-StateFun-hotfix-version-2-2-1-td46239.html
> > > > >
> > > > > On Wed, Nov 4, 2020 at 3:16 PM Tzu-Li (Gordon) Tai <
> > > tzulitai@apache.org>
> > > > > wrote:
> > > > >
> > > > > > > The collect() utils can be picked back, I see no issue with
> that
> > > (it
> > > > is
> > > > > > isolated utilities).
> > > > > >
> > > > > > Just checking on all the requested backports mentioned in this
> > > thread,
> > > > > and
> > > > > > figuring out which ones seem to still be unassigned / open.
> > > > > >
> > > > > > Is someone working on backporting
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/flink/commit/09a7a66b7313fea64817fe960a8da1265b428efc
> > > > > > to release-1.11 at the moment?
> > > > > >
> > > > > > On Wed, Nov 4, 2020 at 1:39 AM Steven Wu <st...@gmail.com>
> > > wrote:
> > > > > >
> > > > > >> @Stephan Ewen <se...@apache.org> yeah, we can do that. don't
> > worry
> > > > > about
> > > > > >> it. your earlier email had the perfect explanation on why file
> > > source
> > > > > >> shouldn't be backported.
> > > > > >>
> > > > > >> On Tue, Nov 3, 2020 at 3:37 AM Stephan Ewen <se...@apache.org>
> > > wrote:
> > > > > >>
> > > > > >> > @Steven would it be possible to initially copy some of the
> code
> > > into
> > > > > the
> > > > > >> > iceberg source and later replace it by a dependency on the
> Flink
> > > > file
> > > > > >> > source?
> > > > > >> >
> > > > > >> > On Mon, Nov 2, 2020 at 8:33 PM Steven Wu <
> stevenz3wu@gmail.com>
> > > > > wrote:
> > > > > >> >
> > > > > >> > > Stephan, thanks a lot for explaining the file connector.
> that
> > > > makes
> > > > > >> > sense.
> > > > > >> > >
> > > > > >> > > I was asking because we were trying to reuse some of the
> > > > > >> implementations
> > > > > >> > in
> > > > > >> > > the file source for Iceberg source. Flink Iceberg source
> lives
> > > in
> > > > > the
> > > > > >> > > Iceberg repo, which is not possible to code against the
> master
> > > > > branch
> > > > > >> of
> > > > > >> > > the Flink code.
> > > > > >> > >
> > > > > >> > > On Mon, Nov 2, 2020 at 3:31 AM Stephan Ewen <
> sewen@apache.org
> > >
> > > > > wrote:
> > > > > >> > >
> > > > > >> > > > Hi Steven!
> > > > > >> > > >
> > > > > >> > > > So far there are no plans to pick back the file system
> > > connector
> > > > > >> code.
> > > > > >> > > This
> > > > > >> > > > is still evolving and not finalized for 1.12, so I don't
> > feel
> > > it
> > > > > is
> > > > > >> a
> > > > > >> > > good
> > > > > >> > > > candidate to be backported.
> > > > > >> > > > However, with the base connector changes backported, you
> > > should
> > > > be
> > > > > >> able
> > > > > >> > > to
> > > > > >> > > > run the file connector code from master against 1.11.3.
> > > > > >> > > >
> > > > > >> > > > The collect() utils can be picked back, I see no issue
> with
> > > that
> > > > > >> (it is
> > > > > >> > > > isolated utilities).
> > > > > >> > > >
> > > > > >> > > > Best,
> > > > > >> > > > Stephan
> > > > > >> > > >
> > > > > >> > > >
> > > > > >> > > > On Mon, Nov 2, 2020 at 3:02 AM Steven Wu <
> > > stevenz3wu@gmail.com>
> > > > > >> wrote:
> > > > > >> > > >
> > > > > >> > > > > Basically, it would be great to get the latest code in
> the
> > > > > >> > > > > flink-connector-files (FLIP-27).
> > > > > >> > > > >
> > > > > >> > > > > On Sat, Oct 31, 2020 at 9:57 AM Steven Wu <
> > > > stevenz3wu@gmail.com
> > > > > >
> > > > > >> > > wrote:
> > > > > >> > > > >
> > > > > >> > > > > > Stephan, it will be great if we can also backport the
> > > > > >> > DataStreamUtils
> > > > > >> > > > > > related commits that help with collecting output from
> > > > > unbounded
> > > > > >> > > > streams.
> > > > > >> > > > > > e.g.
> > > > > >> > > > > >
> > > > > >> > > > > >
> > > > > >> > > > > >
> > > > > >> > > > >
> > > > > >> > > >
> > > > > >> > >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
> https://github.com/apache/flink/commit/09a7a66b7313fea64817fe960a8da1265b428efc
> > > > > >> > > > > >
> > > > > >> > > > > > I tried to copy and paste the code to unblock myself.
> > but
> > > it
> > > > > >> > quickly
> > > > > >> > > > got
> > > > > >> > > > > > into the rabbit hole of more and more code.
> > > > > >> > > > > >
> > > > > >> > > > > > On Fri, Oct 30, 2020 at 11:02 AM Stephan Ewen <
> > > > > sewen@apache.org
> > > > > >> >
> > > > > >> > > > wrote:
> > > > > >> > > > > >
> > > > > >> > > > > >> I have started with backporting the source API
> changes.
> > > > Some
> > > > > >> minor
> > > > > >> > > > > >> conflicts to solve, will need a bit more to finish
> > this.
> > > > > >> > > > > >>
> > > > > >> > > > > >> On Fri, Oct 30, 2020 at 7:25 AM Tzu-Li (Gordon) Tai <
> > > > > >> > > > > tzulitai@apache.org>
> > > > > >> > > > > >> wrote:
> > > > > >> > > > > >>
> > > > > >> > > > > >> > @Stephan Ewen <se...@apache.org>
> > > > > >> > > > > >> > Are there already plans or ongoing efforts for
> > > > backporting
> > > > > >> the
> > > > > >> > > list
> > > > > >> > > > of
> > > > > >> > > > > >> > FLIP-27 changes that you posted?
> > > > > >> > > > > >> >
> > > > > >> > > > > >> > On Thu, Oct 29, 2020 at 7:08 PM Xintong Song <
> > > > > >> > > tonysong820@gmail.com
> > > > > >> > > > >
> > > > > >> > > > > >> > wrote:
> > > > > >> > > > > >> >
> > > > > >> > > > > >> >> Hi folks,
> > > > > >> > > > > >> >>
> > > > > >> > > > > >> >> Just to provide some updates concerning the status
> > on
> > > > the
> > > > > >> > > > > >> >> test instabilities.
> > > > > >> > > > > >> >>
> > > > > >> > > > > >> >> Currently, we have 30 unresolved tickets labeled
> > with
> > > > > >> `Affects
> > > > > >> > > > > Version`
> > > > > >> > > > > >> >> 1.11.x.
> > > > > >> > > > > >> >>
> > > > > >> > > > > >> >>
> > > > > >> > > > > >>
> > > > > >> > > > >
> > > > > >> > > >
> > > > > >> > >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
> https://issues.apache.org/jira/browse/FLINK-19775?filter=12348580&jql=project%20%3D%20FLINK%20AND%20resolution%20%3D%20Unresolved%20AND%20affectedVersion%20in%20(1.11.0%2C%201.11.1%2C%201.11.2%2C%201.11.3)%20AND%20labels%20%3D%20test-stability%20ORDER%20BY%20created%20DESC
> > > > > >> > > > > >> >>
> > > > > >> > > > > >> >> Among the 30 tickets, 11 of them are:
> > > > > >> > > > > >> >> - Have occured in the recent 3 months
> > > > > >> > > > > >> >> - Not confirmed to be pure testability issues
> > > > > >> > > > > >> >> - Not confirmed to be rare condition cases
> > > > > >> > > > > >> >>
> > > > > >> > > > > >> >> It would be nice if someone familiar with these
> > > > components
> > > > > >> can
> > > > > >> > > > take a
> > > > > >> > > > > >> look
> > > > > >> > > > > >> >> into these issues.
> > > > > >> > > > > >> >>
> > > > > >> > > > > >> >> -
> https://issues.apache.org/jira/browse/FLINK-17159
> > > > (ES6)
> > > > > >> > > > > >> >> -
> https://issues.apache.org/jira/browse/FLINK-17912
> > > > > (Kafka)
> > > > > >> > > > > >> >> -
> https://issues.apache.org/jira/browse/FLINK-17949
> > > > > (Kafka)
> > > > > >> > > > > >> >> ⁃
> https://issues.apache.org/jira/browse/FLINK-18444
> > > > > (Kafka)
> > > > > >> > > > > >> >> -
> https://issues.apache.org/jira/browse/FLINK-18634
> > > > > (Kafka)
> > > > > >> > > > > >> >> -
> https://issues.apache.org/jira/browse/FLINK-18648
> > > > > (Kafka)
> > > > > >> > > > > >> >> -
> https://issues.apache.org/jira/browse/FLINK-18807
> > > > > (Kafka)
> > > > > >> > > > > >> >> -
> https://issues.apache.org/jira/browse/FLINK-19369
> > > > > >> > > > (BlobClientTest)
> > > > > >> > > > > >> >> -
> https://issues.apache.org/jira/browse/FLINK-19436
> > > > > (TPCDS)
> > > > > >> > > > > >> >> -
> https://issues.apache.org/jira/browse/FLINK-19690
> > > > > >> > > > (Format/Parquet)
> > > > > >> > > > > >> >> -
> https://issues.apache.org/jira/browse/FLINK-19775
> > > > > >> > > > > >> >> (SystemProcessingTimeServiceTest)
> > > > > >> > > > > >> >>
> > > > > >> > > > > >> >> Thank you~
> > > > > >> > > > > >> >>
> > > > > >> > > > > >> >> Xintong Song
> > > > > >> > > > > >> >>
> > > > > >> > > > > >> >>
> > > > > >> > > > > >> >>
> > > > > >> > > > > >> >> On Thu, Oct 29, 2020 at 10:21 AM Jingsong Li <
> > > > > >> > > > jingsonglee0@gmail.com
> > > > > >> > > > > >
> > > > > >> > > > > >> >> wrote:
> > > > > >> > > > > >> >>
> > > > > >> > > > > >> >> > +1 to backport the FLIP-27 adjustments to
> 1.11.x.
> > > > > >> > > > > >> >> >
> > > > > >> > > > > >> >> > If possible, that would be great. Many people
> are
> > > > > looking
> > > > > >> > > forward
> > > > > >> > > > > to
> > > > > >> > > > > >> the
> > > > > >> > > > > >> >> > FLIP-27 interface, but they don't want to take
> the
> > > > risk
> > > > > to
> > > > > >> > > > upgrade
> > > > > >> > > > > to
> > > > > >> > > > > >> >> 1.12
> > > > > >> > > > > >> >> > (And wait 1.12). After all, 1.11 is a relatively
> > > > stable
> > > > > >> > > version.
> > > > > >> > > > > >> >> >
> > > > > >> > > > > >> >> > Best,
> > > > > >> > > > > >> >> > Jingsong
> > > > > >> > > > > >> >> >
> > > > > >> > > > > >> >> > On Thu, Oct 29, 2020 at 1:24 AM Stephan Ewen <
> > > > > >> > sewen@apache.org
> > > > > >> > > >
> > > > > >> > > > > >> wrote:
> > > > > >> > > > > >> >> >
> > > > > >> > > > > >> >> > > Thanks for starting this.
> > > > > >> > > > > >> >> > >
> > > > > >> > > > > >> >> > > +1 form my side to backport the FLIP-27
> > > adjustments
> > > > to
> > > > > >> > > 1.11.x.
> > > > > >> > > > > >> >> > >
> > > > > >> > > > > >> >> > > There were quite a few changes, and I think we
> > > need
> > > > to
> > > > > >> > > > > cherry-pick
> > > > > >> > > > > >> >> them
> > > > > >> > > > > >> >> > all
> > > > > >> > > > > >> >> > > to not get some inconsistent mix of changes
> and
> > > many
> > > > > >> merge
> > > > > >> > > > > >> conflicts.
> > > > > >> > > > > >> >> > > I made a list below of what we need to add to
> > > > > >> > "release-1.11".
> > > > > >> > > > > >> >> > >
> > > > > >> > > > > >> >> > > *#### Core Source API Changes to backport (in
> > > > REVERSE
> > > > > >> > order)*
> > > > > >> > > > > >> >> > >
> > > > > >> > > > > >> >> > >   (Use: "git log
> > > > > >> > > > > >> >> > >
> > > > > >> > > >
> > > flink-core/src/main/java/org/apache/flink/api/connector/source")
> > > > > >> > > > > >> >> > >
> > > > > >> > > > > >> >> > > commit
> 162c072e9265a7b6dd9d6f5459eb7974091c4c4e
> > > > > >> > > > > >> >> > > [FLINK-19492][core] Consolidate Source Events
> > > > between
> > > > > >> > Source
> > > > > >> > > > API
> > > > > >> > > > > >> and
> > > > > >> > > > > >> >> > Split
> > > > > >> > > > > >> >> > > Reader API
> > > > > >> > > > > >> >> > >
> > > > > >> > > > > >> >> > > commit
> 428fbbad85379f71a9fbf1a1ab3e3dcfa9006436
> > > > > >> > > > > >> >> > > [FLINK-19205][core] Add access to
> configuration
> > > and
> > > > > >> > hostname
> > > > > >> > > in
> > > > > >> > > > > the
> > > > > >> > > > > >> >> > > SourceReaderContext
> > > > > >> > > > > >> >> > >
> > > > > >> > > > > >> >> > > commit
> 723e1790744ddc78d19c7c978442af1383f38d33
> > > > > >> > > > > >> >> > > [hotfix][core] Add to Source Enumerator
> > > convenience
> > > > > >> methods
> > > > > >> > > to
> > > > > >> > > > > >> assign
> > > > > >> > > > > >> >> > > single split
> > > > > >> > > > > >> >> > >
> > > > > >> > > > > >> >> > > *#### Connector Base Changes to Backport (in
> > > REVERSE
> > > > > >> > order)*
> > > > > >> > > > > >> >> > >
> > > > > >> > > > > >> >> > >   (Use: "git log
> > > > > flink-connectors/flink-connector-base")
> > > > > >> > > > > >> >> > >
> > > > > >> > > > > >> >> > > commit
> 401f56fe9d6b0271260edf9787cdcbfe4d03874d
> > > > > >> > > > > >> >> > > [FLINK-19427][FLINK-19489][tests] Fix test
> > > > conditions
> > > > > >> for
> > > > > >> > > > > >> >> > >
> > > > > 'SplitFetcherTest.testNotifiesWhenGoingIdleConcurrent()'
> > > > > >> > > > > >> >> > >
> > > > > >> > > > > >> >> > > commit
> 68c5c2ff779d82a1ff81ffaf60d8a1b283797db1
> > > > > >> > > > > >> >> > > [FLINK-19448][connector base] Explicitly check
> > for
> > > > > >> > > un-expected
> > > > > >> > > > > >> >> condition
> > > > > >> > > > > >> >> > > that would leave an inconsistent state
> > > > > >> > > > > >> >> > >
> > > > > >> > > > > >> >> > > commit
> 162c072e9265a7b6dd9d6f5459eb7974091c4c4e
> > > > > >> > > > > >> >> > > [FLINK-19492][core] Consolidate Source Events
> > > > between
> > > > > >> > Source
> > > > > >> > > > API
> > > > > >> > > > > >> and
> > > > > >> > > > > >> >> > Split
> > > > > >> > > > > >> >> > > Reader API
> > > > > >> > > > > >> >> > >
> > > > > >> > > > > >> >> > > commit
> c1ca7a4c7c21ec8868c14cdddd43c559625b794c
> > > > > >> > > > > >> >> > > [refactor][tests] Move some source test utils
> > from
> > > > > >> > > > > >> >> flink-connector-base
> > > > > >> > > > > >> >> > to
> > > > > >> > > > > >> >> > > flink-core
> > > > > >> > > > > >> >> > >
> > > > > >> > > > > >> >> > > commit
> ee5c4c211c35c70d28252363bbc8400453609977
> > > > > >> > > > > >> >> > > [FLINK-19251][connectors] Avoid confusing
> queue
> > > > > >> handling in
> > > > > >> > > > > >> >> > > "SplitReader.handleSplitsChanges()"
> > > > > >> > > > > >> >> > >
> > > > > >> > > > > >> >> > > commit
> 5abef56b2bf85bcac786f6b16b6899b6cced7176
> > > > > >> > > > > >> >> > > [FLINK-19250][connectors] Fix error
> propagation
> > in
> > > > > >> > connector
> > > > > >> > > > base
> > > > > >> > > > > >> >> > > (SplitFetcherManager).
> > > > > >> > > > > >> >> > >
> > > > > >> > > > > >> >> > > commit
> 8fcca837c55a9216595ee4c03038b52747098dbb
> > > > > >> > > > > >> >> > > [hotfix][connectors] Improve JavaDocs for
> > > > > >> > > > > >> SingleThreadFetcherManager
> > > > > >> > > > > >> >> > >
> > > > > >> > > > > >> >> > > commit
> 4700bb5dde3303cbe98882f6beb7379425717b01
> > > > > >> > > > > >> >> > > [FLINK-19225][connectors] Various small
> > > improvements
> > > > > to
> > > > > >> > > > > >> >> SourceReaderBase
> > > > > >> > > > > >> >> > > (part 2)
> > > > > >> > > > > >> >> > >
> > > > > >> > > > > >> >> > > commit
> 12261c6b7ed6478a9b9f6a69cb58246b83cab9b7
> > > > > >> > > > > >> >> > > [FLINK-17393][connectors] (follow-up) Wakeup
> the
> > > > > >> > > SplitFetchers
> > > > > >> > > > > more
> > > > > >> > > > > >> >> > > elegantly.
> > > > > >> > > > > >> >> > >
> > > > > >> > > > > >> >> > > commit
> c60aaff0249bfd6b5871b7f82e03efc487a54d6b
> > > > > >> > > > > >> >> > > [hotfix][tests] Extend test coverage for
> > > > > >> > > > > >> FutureCompletingBlockingQueue
> > > > > >> > > > > >> >> > >
> > > > > >> > > > > >> >> > > commit
> cef8a587d7fd2fe64cc644da5ed095d82e46f631
> > > > > >> > > > > >> >> > > [FLINK-19245][connectors] Set default capacity
> > for
> > > > > >> > > > > >> >> > > FutureCompletingBlockingQueue.
> > > > > >> > > > > >> >> > >
> > > > > >> > > > > >> >> > > commit
> 4ea95782b4c6a2538153d4d16ad3f4839c7de0fb
> > > > > >> > > > > >> >> > > [FLINK-19223][connectors] Simplify
> Availability
> > > > Future
> > > > > >> > Model
> > > > > >> > > in
> > > > > >> > > > > >> Base
> > > > > >> > > > > >> >> > > Connector
> > > > > >> > > > > >> >> > >
> > > > > >> > > > > >> >> > > commit
> 511857049ba30c8ff0ee56da551fa4a479dc583e
> > > > > >> > > > > >> >> > > [FLINK-18128][connectors] Ensure idle split
> > > fetchers
> > > > > >> lead
> > > > > >> > to
> > > > > >> > > > > >> >> availability
> > > > > >> > > > > >> >> > > notifications.
> > > > > >> > > > > >> >> > >
> > > > > >> > > > > >> >> > > commit
> a8206467af0830dcb89623ea068b5ca3b3450c92
> > > > > >> > > > > >> >> > > [refactor][core] Eagerly initialize the
> > FetchTask
> > > to
> > > > > >> > support
> > > > > >> > > > > proper
> > > > > >> > > > > >> >> unit
> > > > > >> > > > > >> >> > > testing
> > > > > >> > > > > >> >> > >
> > > > > >> > > > > >> >> > > commit
> 3b2f54bcb437f98e6137c904045cc51072b5c06b
> > > > > >> > > > > >> >> > > [hotfix][tests] Move constants in
> > SplitFetcherTest
> > > > > >> relevant
> > > > > >> > > to
> > > > > >> > > > > only
> > > > > >> > > > > >> >> one
> > > > > >> > > > > >> >> > > test into test method
> > > > > >> > > > > >> >> > >
> > > > > >> > > > > >> >> > > commit
> d7625760a75a508bf05bcddc380bb4d62ee1743e
> > > > > >> > > > > >> >> > > [FLINK-19225][connectors] Various small
> > > improvements
> > > > > to
> > > > > >> > > > > >> >> SourceReaderBase
> > > > > >> > > > > >> >> > >
> > > > > >> > > > > >> >> > > commit
> a5b0d3297748c1be47ad579a88f24df2255a8df1
> > > > > >> > > > > >> >> > > [FLINK-17393][connectors] Wakeup the
> > SplitFetchers
> > > > > more
> > > > > >> > > > > elegantly.
> > > > > >> > > > > >> >> > >
> > > > > >> > > > > >> >> > > commit
> f42a3ebc3e81a034b7221a803c153636fef34903
> > > > > >> > > > > >> >> > > [FLINK-18680][connectors] Make connector base
> > > > > >> > > > RecordsWithSplitIds
> > > > > >> > > > > >> more
> > > > > >> > > > > >> >> > > lightweight.
> > > > > >> > > > > >> >> > >
> > > > > >> > > > > >> >> > > commit
> e3d273de822b085183d09b275a445879ff94b350
> > > > > >> > > > > >> >> > > [FLINK-19162][connectors] Add 'recycle()' to
> the
> > > > > >> > > > > >> RecordsWithSplitIds
> > > > > >> > > > > >> >> to
> > > > > >> > > > > >> >> > > support reuse of heavy objects.
> > > > > >> > > > > >> >> > >
> > > > > >> > > > > >> >> > > commit
> 8ebc464c2520453a70001cd712abc8dee6ee89e0
> > > > > >> > > > > >> >> > > [hotfix][testing] Add a set of parameterizable
> > > > testing
> > > > > >> > mocks
> > > > > >> > > > for
> > > > > >> > > > > >> the
> > > > > >> > > > > >> >> > Split
> > > > > >> > > > > >> >> > > Reader API
> > > > > >> > > > > >> >> > >
> > > > > >> > > > > >> >> > > commit
> 930a07438be1185388d7150640f294dfe2a5d378
> > > > > >> > > > > >> >> > > [hotfix][connectors] Add RequestSplitEvent to
> > > > > >> > > > > >> 'flink-connector-base'
> > > > > >> > > > > >> >> > >
> > > > > >> > > > > >> >> > > commit
> e945ce8a933bc378844782f784ca473c767ca159
> > > > > >> > > > > >> >> > > [refactor][connectors] Backport of the
> > > > connector-base
> > > > > >> > > exception
> > > > > >> > > > > >> >> handling
> > > > > >> > > > > >> >> > > from the Kafka Connector Pull Request
> > > > > >> > > > > >> >> > >
> > > > > >> > > > > >> >> > >
> > > > > >> > > > > >> >> > > On Wed, Oct 28, 2020 at 5:49 PM Steven Wu <
> > > > > >> > > > stevenz3wu@gmail.com>
> > > > > >> > > > > >> >> wrote:
> > > > > >> > > > > >> >> > >
> > > > > >> > > > > >> >> > > > I would love to see this FLIP-27 source
> > > interface
> > > > > >> > > improvement
> > > > > >> > > > > [1]
> > > > > >> > > > > >> >> made
> > > > > >> > > > > >> >> > to
> > > > > >> > > > > >> >> > > > 1.11.3.
> > > > > >> > > > > >> >> > > >
> > > > > >> > > > > >> >> > > > [1]
> > > > > https://issues.apache.org/jira/browse/FLINK-19698
> > > > > >> > > > > >> >> > > >
> > > > > >> > > > > >> >> > > > On Wed, Oct 28, 2020 at 12:32 AM Tzu-Li
> > (Gordon)
> > > > > Tai <
> > > > > >> > > > > >> >> > > tzulitai@apache.org>
> > > > > >> > > > > >> >> > > > wrote:
> > > > > >> > > > > >> >> > > >
> > > > > >> > > > > >> >> > > > > Thanks for the replies so far!
> > > > > >> > > > > >> >> > > > >
> > > > > >> > > > > >> >> > > > > Just to provide a brief update on the
> status
> > > of
> > > > > >> > blockers
> > > > > >> > > > for
> > > > > >> > > > > >> >> 1.11.3
> > > > > >> > > > > >> >> > so
> > > > > >> > > > > >> >> > > > far:
> > > > > >> > > > > >> >> > > > >
> > > > > >> > > > > >> >> > > > >
> > > > > >> > > > > >> >> > > > > *PR opened, pending reviewer*-
> [FLINK-19717]
> > > > > >> > > > > >> >> > SourceReaderBase.pollNext
> > > > > >> > > > > >> >> > > > may
> > > > > >> > > > > >> >> > > > > return END_OF_INPUT if SplitReader.fetch
> > > throws
> > > > (
> > > > > >> > > > > >> >> > > > >
> https://github.com/apache/flink/pull/13776)
> > > > > >> > > > > >> >> > > > >
> > > > > >> > > > > >> >> > > > > *PR opened, reviewed + close to being
> > merged*
> > > > > >> > > > > >> >> > > > > - [FLINK-19741] Timer service should skip
> > > > > restoring
> > > > > >> > from
> > > > > >> > > > raw
> > > > > >> > > > > >> keyed
> > > > > >> > > > > >> >> > > stream
> > > > > >> > > > > >> >> > > > > if it isn't the writer (
> > > > > >> > > > > >> >> https://github.com/apache/flink/pull/13761)
> > > > > >> > > > > >> >> > > > > - [FLINK-19748] Raw keyed stream key group
> > > > > iterator
> > > > > >> > > should
> > > > > >> > > > be
> > > > > >> > > > > >> >> > skipping
> > > > > >> > > > > >> >> > > > > unwritten key groups (
> > > > > >> > > > > >> https://github.com/apache/flink/pull/13772)
> > > > > >> > > > > >> >> > > > >
> > > > > >> > > > > >> >> > > > > *Merged*
> > > > > >> > > > > >> >> > > > > - [FLINK-19154] Application mode deletes
> HA
> > > data
> > > > > in
> > > > > >> > case
> > > > > >> > > of
> > > > > >> > > > > >> >> suspended
> > > > > >> > > > > >> >> > > > > ZooKeeper connection
> > > > > >> > > > > >> >> > > > > - [FLINK-19569] Upgrade ICU4J to 67.1+
> > > > > >> > > > > >> >> > > > >
> > > > > >> > > > > >> >> > > > > Right now as it seems, progress is mainly
> > > > blocked
> > > > > >> on a
> > > > > >> > > > > reviewer
> > > > > >> > > > > >> >> for
> > > > > >> > > > > >> >> > > > > FLINK-19717.
> > > > > >> > > > > >> >> > > > > Meanwhile, Xintong is keeping an eye on
> test
> > > > > >> > > instabilities
> > > > > >> > > > > [1]
> > > > > >> > > > > >> to
> > > > > >> > > > > >> >> see
> > > > > >> > > > > >> >> > > if
> > > > > >> > > > > >> >> > > > > there are any fixes that should be applied
> > to
> > > > > >> > > > `release-1.11`.
> > > > > >> > > > > >> >> > > > >
> > > > > >> > > > > >> >> > > > > This is also a reminder, that if there are
> > > other
> > > > > >> > blockers
> > > > > >> > > > > that
> > > > > >> > > > > >> we
> > > > > >> > > > > >> >> > need
> > > > > >> > > > > >> >> > > to
> > > > > >> > > > > >> >> > > > > be aware of, or a need to re-establish
> > > estimated
> > > > > >> time
> > > > > >> > for
> > > > > >> > > > > >> getting
> > > > > >> > > > > >> >> > fixes
> > > > > >> > > > > >> >> > > > in
> > > > > >> > > > > >> >> > > > > and delay the RC for 1.11.3, please do let
> > us
> > > > > know!
> > > > > >> > > > > >> >> > > > >
> > > > > >> > > > > >> >> > > > > Cheers,
> > > > > >> > > > > >> >> > > > > Gordon
> > > > > >> > > > > >> >> > > > >
> > > > > >> > > > > >> >> > > > > [1]
> > > > > >> > > > > >> >> > > > >
> > > > > >> > > > > >> >> > > > >
> > > > > >> > > > > >> >> > > >
> > > > > >> > > > > >> >> > >
> > > > > >> > > > > >> >> >
> > > > > >> > > > > >> >>
> > > > > >> > > > > >>
> > > > > >> > > > >
> > > > > >> > > >
> > > > > >> > >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
> https://issues.apache.org/jira/issues/?jql=project%20%3D%20FLINK%20AND%20status%20in%20(Open%2C%20%22In%20Progress%22%2C%20Reopened)%20AND%20affectedVersion%20in%20(1.11.0%2C%201.11.1%2C%201.11.2)%20AND%20labels%20%3D%20test-stability
> > > > > >> > > > > >> >> > > > >
> > > > > >> > > > > >> >> > > > > On Mon, Oct 26, 2020 at 9:43 PM Kostas
> > > Kloudas <
> > > > > >> > > > > >> >> kkloudas@apache.org>
> > > > > >> > > > > >> >> > > > > wrote:
> > > > > >> > > > > >> >> > > > >
> > > > > >> > > > > >> >> > > > > > +1 for releasing Flink 1.11.3 as it
> > > contains a
> > > > > >> number
> > > > > >> > > of
> > > > > >> > > > > >> >> important
> > > > > >> > > > > >> >> > > > > > fixes and thanks Gordon and Xintong for
> > > > > >> volunteering.
> > > > > >> > > > > >> >> > > > > >
> > > > > >> > > > > >> >> > > > > > Cheers,
> > > > > >> > > > > >> >> > > > > > Kostas
> > > > > >> > > > > >> >> > > > > >
> > > > > >> > > > > >> >> > > > > > On Mon, Oct 26, 2020 at 4:37 AM Yu Li <
> > > > > >> > > carp84@gmail.com>
> > > > > >> > > > > >> wrote:
> > > > > >> > > > > >> >> > > > > > >
> > > > > >> > > > > >> >> > > > > > > +1 for releasing Flink 1.11.3, and
> > thanks
> > > > > Gordon
> > > > > >> > and
> > > > > >> > > > > >> Xintong
> > > > > >> > > > > >> >> for
> > > > > >> > > > > >> >> > > > > > > volunteering as our release managers.
> > > > > >> > > > > >> >> > > > > > >
> > > > > >> > > > > >> >> > > > > > > Best Regards,
> > > > > >> > > > > >> >> > > > > > > Yu
> > > > > >> > > > > >> >> > > > > > >
> > > > > >> > > > > >> >> > > > > > >
> > > > > >> > > > > >> >> > > > > > > On Mon, 26 Oct 2020 at 09:45, Xintong
> > > Song <
> > > > > >> > > > > >> >> > tonysong820@gmail.com>
> > > > > >> > > > > >> >> > > > > > wrote:
> > > > > >> > > > > >> >> > > > > > >
> > > > > >> > > > > >> >> > > > > > > > Thanks Gordan for starting this
> > > > discussion.
> > > > > >> > > > > >> >> > > > > > > > My pleasure to help with the release
> > > > > process.
> > > > > >> > > > > >> >> > > > > > > >
> > > > > >> > > > > >> >> > > > > > > > Thank you~
> > > > > >> > > > > >> >> > > > > > > >
> > > > > >> > > > > >> >> > > > > > > > Xintong Song
> > > > > >> > > > > >> >> > > > > > > >
> > > > > >> > > > > >> >> > > > > > > >
> > > > > >> > > > > >> >> > > > > > > >
> > > > > >> > > > > >> >> > > > > > > > On Fri, Oct 23, 2020 at 11:29 PM
> Till
> > > > > >> Rohrmann <
> > > > > >> > > > > >> >> > > > trohrmann@apache.org
> > > > > >> > > > > >> >> > > > > >
> > > > > >> > > > > >> >> > > > > > > > wrote:
> > > > > >> > > > > >> >> > > > > > > >
> > > > > >> > > > > >> >> > > > > > > > > Thanks for starting this
> discussion
> > > > > Gordon.
> > > > > >> > There
> > > > > >> > > > are
> > > > > >> > > > > >> over
> > > > > >> > > > > >> >> > 100
> > > > > >> > > > > >> >> > > > > issues
> > > > > >> > > > > >> >> > > > > > > > > which are fixed for 1.11.3. Hence
> +1
> > > > for a
> > > > > >> > > soonish
> > > > > >> > > > > >> 1.11.3
> > > > > >> > > > > >> >> > > > release.
> > > > > >> > > > > >> >> > > > > > Thanks
> > > > > >> > > > > >> >> > > > > > > > > for volunteering as our release
> > > managers
> > > > > >> Gordon
> > > > > >> > > and
> > > > > >> > > > > >> >> Xintong!
> > > > > >> > > > > >> >> > > > > > > > >
> > > > > >> > > > > >> >> > > > > > > > > Cheers,
> > > > > >> > > > > >> >> > > > > > > > > Till
> > > > > >> > > > > >> >> > > > > > > > >
> > > > > >> > > > > >> >> > > > > > > > > On Fri, Oct 23, 2020 at 5:02 PM
> > Tzu-Li
> > > > > >> (Gordon)
> > > > > >> > > > Tai <
> > > > > >> > > > > >> >> > > > > > tzulitai@apache.org
> > > > > >> > > > > >> >> > > > > > > > >
> > > > > >> > > > > >> >> > > > > > > > > wrote:
> > > > > >> > > > > >> >> > > > > > > > >
> > > > > >> > > > > >> >> > > > > > > > >> Hi,
> > > > > >> > > > > >> >> > > > > > > > >>
> > > > > >> > > > > >> >> > > > > > > > >> Xintong and I would like to
> start a
> > > > > >> discussion
> > > > > >> > > for
> > > > > >> > > > > >> >> releasing
> > > > > >> > > > > >> >> > > > Flink
> > > > > >> > > > > >> >> > > > > > > > 1.11.3
> > > > > >> > > > > >> >> > > > > > > > >> soon.
> > > > > >> > > > > >> >> > > > > > > > >>
> > > > > >> > > > > >> >> > > > > > > > >> It seems like we already have a
> few
> > > > > >> pressing
> > > > > >> > > > issues
> > > > > >> > > > > >> that
> > > > > >> > > > > >> >> > needs
> > > > > >> > > > > >> >> > > > to
> > > > > >> > > > > >> >> > > > > be
> > > > > >> > > > > >> >> > > > > > > > >> included in a new hotfix release:
> > > > > >> > > > > >> >> > > > > > > > >>
> > > > > >> > > > > >> >> > > > > > > > >>    - Heap-based timers’ restore
> > > > behaviour
> > > > > >> is
> > > > > >> > > > > causing a
> > > > > >> > > > > >> >> > > critical
> > > > > >> > > > > >> >> > > > > > recovery
> > > > > >> > > > > >> >> > > > > > > > >>    issue for StateFun [1] [2]
> [3].
> > > > > >> > > > > >> >> > > > > > > > >>    - There are several robustness
> > > > issues
> > > > > >> for
> > > > > >> > the
> > > > > >> > > > > >> FLIP-27
> > > > > >> > > > > >> >> new
> > > > > >> > > > > >> >> > > > > source
> > > > > >> > > > > >> >> > > > > > API,
> > > > > >> > > > > >> >> > > > > > > > >>    such as [4]. We already have
> > some
> > > > > users
> > > > > >> > using
> > > > > >> > > > the
> > > > > >> > > > > >> >> FLIP-27
> > > > > >> > > > > >> >> > > API
> > > > > >> > > > > >> >> > > > > > with
> > > > > >> > > > > >> >> > > > > > > > >> 1.11.x,
> > > > > >> > > > > >> >> > > > > > > > >>    so it would be important to
> get
> > > > those
> > > > > >> fixes
> > > > > >> > > in
> > > > > >> > > > > for
> > > > > >> > > > > >> >> 1.11.x
> > > > > >> > > > > >> >> > > as
> > > > > >> > > > > >> >> > > > > > well.
> > > > > >> > > > > >> >> > > > > > > > >>
> > > > > >> > > > > >> >> > > > > > > > >> Apart from the issues that are
> > > already
> > > > > >> marked
> > > > > >> > as
> > > > > >> > > > > >> blocker
> > > > > >> > > > > >> >> for
> > > > > >> > > > > >> >> > > > > 1.11.3
> > > > > >> > > > > >> >> > > > > > in
> > > > > >> > > > > >> >> > > > > > > > our
> > > > > >> > > > > >> >> > > > > > > > >> JIRA [5], please let us know in
> > this
> > > > > >> thread if
> > > > > >> > > > there
> > > > > >> > > > > >> is
> > > > > >> > > > > >> >> > > already
> > > > > >> > > > > >> >> > > > > > ongoing
> > > > > >> > > > > >> >> > > > > > > > >> work for other important fixes
> that
> > > we
> > > > > >> should
> > > > > >> > > try
> > > > > >> > > > to
> > > > > >> > > > > >> >> > include.
> > > > > >> > > > > >> >> > > > > > > > >>
> > > > > >> > > > > >> >> > > > > > > > >> Xintong and I would like to
> > volunteer
> > > > for
> > > > > >> > > managing
> > > > > >> > > > > >> this
> > > > > >> > > > > >> >> > > release,
> > > > > >> > > > > >> >> > > > > and
> > > > > >> > > > > >> >> > > > > > > > will
> > > > > >> > > > > >> >> > > > > > > > >> try to communicate the priority
> of
> > > > > pending
> > > > > >> > > > blockers
> > > > > >> > > > > >> over
> > > > > >> > > > > >> >> the
> > > > > >> > > > > >> >> > > > next
> > > > > >> > > > > >> >> > > > > > few
> > > > > >> > > > > >> >> > > > > > > > >> days.
> > > > > >> > > > > >> >> > > > > > > > >> Since the aforementioned issues
> are
> > > > quite
> > > > > >> > > > critical,
> > > > > >> > > > > >> we’d
> > > > > >> > > > > >> >> > like
> > > > > >> > > > > >> >> > > to
> > > > > >> > > > > >> >> > > > > aim
> > > > > >> > > > > >> >> > > > > > > > >> for a *feature
> > > > > >> > > > > >> >> > > > > > > > >> freeze by the end of next week
> > (Oct.
> > > > > 30th)*
> > > > > >> > and
> > > > > >> > > > > start
> > > > > >> > > > > >> the
> > > > > >> > > > > >> >> > > > release
> > > > > >> > > > > >> >> > > > > > voting
> > > > > >> > > > > >> >> > > > > > > > >> process the week after.
> > > > > >> > > > > >> >> > > > > > > > >> If that is too short of a notice
> > and
> > > > you
> > > > > >> might
> > > > > >> > > > need
> > > > > >> > > > > >> more
> > > > > >> > > > > >> >> > time,
> > > > > >> > > > > >> >> > > > > > please
> > > > > >> > > > > >> >> > > > > > > > let
> > > > > >> > > > > >> >> > > > > > > > >> us know!
> > > > > >> > > > > >> >> > > > > > > > >>
> > > > > >> > > > > >> >> > > > > > > > >> Cheers,
> > > > > >> > > > > >> >> > > > > > > > >> Gordon
> > > > > >> > > > > >> >> > > > > > > > >>
> > > > > >> > > > > >> >> > > > > > > > >> [1]
> > > > > >> > > > > https://issues.apache.org/jira/browse/FLINK-19692
> > > > > >> > > > > >> >> > > > > > > > >> [2]
> > > > > >> > > > > https://issues.apache.org/jira/browse/FLINK-19741
> > > > > >> > > > > >> >> > > > > > > > >> [3]
> > > > > >> > > > > https://issues.apache.org/jira/browse/FLINK-19748
> > > > > >> > > > > >> >> > > > > > > > >> [4]
> > > > > >> > > > > https://issues.apache.org/jira/browse/FLINK-19717
> > > > > >> > > > > >> >> > > > > > > > >> [5]
> > > > > >> > > > > >> >> > > > > > > > >>
> > > > > >> > > > > >> >> > > > > > > > >>
> > > > > >> > > > > >> >> > > > > > > >
> > > > > >> > > > > >> >> > > > > >
> > > > > >> > > > > >> >> > > > >
> > > > > >> > > > > >> >> > > >
> > > > > >> > > > > >> >> > >
> > > > > >> > > > > >> >> >
> > > > > >> > > > > >> >>
> > > > > >> > > > > >>
> > > > > >> > > > >
> > > > > >> > > >
> > > > > >> > >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
> https://issues.apache.org/jira/issues/?jql=project%20%3D%20FLINK%20AND%20priority%20%3D%20Blocker%20AND%20fixVersion%20%3D%201.11.3
> > > > > >> > > > > >> >> > > > > > > > >>
> > > > > >> > > > > >> >> > > > > > > > >
> > > > > >> > > > > >> >> > > > > > > >
> > > > > >> > > > > >> >> > > > > >
> > > > > >> > > > > >> >> > > > >
> > > > > >> > > > > >> >> > > >
> > > > > >> > > > > >> >> > >
> > > > > >> > > > > >> >> >
> > > > > >> > > > > >> >> >
> > > > > >> > > > > >> >> > --
> > > > > >> > > > > >> >> > Best, Jingsong Lee
> > > > > >> > > > > >> >> >
> > > > > >> > > > > >> >>
> > > > > >> > > > > >> >
> > > > > >> > > > > >>
> > > > > >> > > > > >
> > > > > >> > > > >
> > > > > >> > > >
> > > > > >> > >
> > > > > >> >
> > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] Releasing Apache Flink 1.11.3

Posted by Till Rohrmann <tr...@apache.org>.
Yes, I've downgraded FLINK-19816 to critical.

Cheers,
Till

On Mon, Nov 9, 2020 at 10:19 AM Xintong Song <to...@gmail.com> wrote:

> Thanks for the notice, Till.
>
> I just checked and found FLINK-20033 is already fixed. Shall we also
> downgrade FLINK-19816 to `Critical`?
>
> Thank you~
>
> Xintong Song
>
>
>
> On Mon, Nov 9, 2020 at 4:42 PM Till Rohrmann <tr...@apache.org> wrote:
>
> > I would like to bring one more critical issue to your attention which is
> > FLINK-20033 [1]. I believe that this issue is actually causing what has
> > been reported in FLINK-19816 [2]. I hope to have it fixed by the end of
> > today. Once FLINK-20033 is fixed, I think that we don't have to block the
> > release on FLINK-19816.
> >
> > [1] https://issues.apache.org/jira/browse/FLINK-20033
> > [2] https://issues.apache.org/jira/browse/FLINK-19816
> >
> > Cheers,
> > Till
> >
> > On Mon, Nov 9, 2020 at 4:05 AM Xintong Song <to...@gmail.com>
> wrote:
> >
> > > Hi devs,
> > >
> > > I'd like to provide an update on the progress of preparing release
> > 1.11.3.
> > >
> > > *Blockers*
> > > We currently have 3 remaining blockers. (3 resolved and 1 emerged
> > compared
> > > to last week)
> > >
> > > - [FLINK-19698] Add close() method and onCheckpointComplete() to the
> > > Source.
> > > The issue has been fixed on the master branch. It's currently blocked
> on
> > > the FLIP-27 backportings to backport it to the 1.11 branch.
> > >
> > > - [FLINK-19717] SourceReaderBase.pollNext may return END_OF_INPUT if
> > > SplitReader.fetch throws
> > > A PR has been opened and reviewed. From the discussions on the PR, it
> > looks
> > > close to mergeable.
> > >
> > > - [FLINK-19816] Flink restored from a wrong checkpoint (a very old one
> > and
> > > not the last completed one)
> > > This is a newly emerged blocker and Matthias is working on it.
> > >
> > > *Test Instabilities*
> > > We currently have 27 test instabilities[1].
> > > AFAIK, none of them are as serious as to block the 1.11.3 release.
> > >
> > > *FLIP-27 Backprotings*
> > >
> > > I noticed that there's no jira issues opened on the FLIP-27 backporting
> > > efforts, which is part of the major efforts planned for the 1.11.3
> > release,
> > > making it hard to track the progress.
> > >
> > >
> > > @Stephan and @Becket, could you please share the updates on the
> > backporting
> > > efforts? How is the progress and when are the efforts expected to be
> > > finished? It would be appreciated and helpful if we can have a jira
> > ticket
> > > to track the backporing progress publicly.
> > >
> > >
> > > Thank you~
> > >
> > > Xintong Song
> > >
> > >
> > > [1]
> > >
> > >
> >
> https://issues.apache.org/jira/browse/FLINK-16908?filter=12348580&jql=project%20%3D%20FLINK%20AND%20resolution%20%3D%20Unresolved%20AND%20affectedVersion%20in%20(1.11.0%2C%201.11.1%2C%201.11.2%2C%201.11.3)%20AND%20labels%20%3D%20test-stability%20ORDER%20BY%20updated%20DESC%2C%20created%20DESC
> > >
> > > On Wed, Nov 4, 2020 at 3:58 PM Tzu-Li (Gordon) Tai <
> tzulitai@apache.org>
> > > wrote:
> > >
> > > > Thanks for the efforts so far with pushing for 1.11.3.
> > > >
> > > > I'd like to provide a mid-week update on how we're looking with all
> the
> > > > pending blockers and backports:
> > > >
> > > > *Blockers**:*
> > > >
> > > > Currently all blockers either have an open PR, or have been merged.
> > I'll
> > > > highlight below the blockers *that still require reviewing efforts
> for
> > > them
> > > > to move forward:*
> > > >
> > > > - [FLINK-19909] Flink application in attach mode could not terminate
> > when
> > > > the only job is canceled. PR:
> > https://github.com/apache/flink/pull/13911
> > > > - [FLINK-19717] SourceReaderBase.pollNext may return END_OF_INPUT if
> > > > SplitReader.fetch throws. PR:
> > https://github.com/apache/flink/pull/13776
> > > >
> > > > The above PRs currently have no reviews at all yet. They seem to
> > already
> > > > have designated reviewers.
> > > >
> > > >
> > > > *Backports:*
> > > > - FLIP-27 robustness improvement backports: Stephan is currently
> > working
> > > on
> > > > backporting several FLIP-27 changes. There are no PRs yet for the
> > > > backported changes.
> > > > - DataStreamUtils.collect() refactorings backport: Steven Wu
> mentioned
> > to
> > > > backport this, but AFAIK this isn't assigned to anyone yet.
> > > >
> > > > *ETAs*:
> > > >
> > > > I'd like to request ETAs for the remaining backports, to prevent a
> > creep
> > > in
> > > > the scope of this bugfix release.
> > > >
> > > > We already have Flink users that would benefit from fixes that have
> > been
> > > > merged to release-1.11 already, so technically speaking the backports
> > > > should be considered "nice-to-have" (to the best of my knowledge of
> the
> > > > changes) and could potentially be moved to a follow-up 1.11.4.
> > > > Most notably, the Stateful Functions project is already waiting on
> > Flink
> > > > 1.11.3 to address critical recovery issues (please see the StateFun
> > 2.2.1
> > > > release discussion thread [1]).
> > > >
> > > > @Stephan Ewen <se...@apache.org> @Becket Qin <be...@gmail.com>
> > > could
> > > > you provide an ETA for the FLIP-27 backports? It would help to get a
> > > better
> > > > estimate to decide how we proceed here.
> > > >
> > > > Cheers,
> > > > Gordon
> > > >
> > > > [1]
> > > >
> > > >
> > >
> >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Releasing-StateFun-hotfix-version-2-2-1-td46239.html
> > > >
> > > > On Wed, Nov 4, 2020 at 3:16 PM Tzu-Li (Gordon) Tai <
> > tzulitai@apache.org>
> > > > wrote:
> > > >
> > > > > > The collect() utils can be picked back, I see no issue with that
> > (it
> > > is
> > > > > isolated utilities).
> > > > >
> > > > > Just checking on all the requested backports mentioned in this
> > thread,
> > > > and
> > > > > figuring out which ones seem to still be unassigned / open.
> > > > >
> > > > > Is someone working on backporting
> > > > >
> > > >
> > >
> >
> https://github.com/apache/flink/commit/09a7a66b7313fea64817fe960a8da1265b428efc
> > > > > to release-1.11 at the moment?
> > > > >
> > > > > On Wed, Nov 4, 2020 at 1:39 AM Steven Wu <st...@gmail.com>
> > wrote:
> > > > >
> > > > >> @Stephan Ewen <se...@apache.org> yeah, we can do that. don't
> worry
> > > > about
> > > > >> it. your earlier email had the perfect explanation on why file
> > source
> > > > >> shouldn't be backported.
> > > > >>
> > > > >> On Tue, Nov 3, 2020 at 3:37 AM Stephan Ewen <se...@apache.org>
> > wrote:
> > > > >>
> > > > >> > @Steven would it be possible to initially copy some of the code
> > into
> > > > the
> > > > >> > iceberg source and later replace it by a dependency on the Flink
> > > file
> > > > >> > source?
> > > > >> >
> > > > >> > On Mon, Nov 2, 2020 at 8:33 PM Steven Wu <st...@gmail.com>
> > > > wrote:
> > > > >> >
> > > > >> > > Stephan, thanks a lot for explaining the file connector. that
> > > makes
> > > > >> > sense.
> > > > >> > >
> > > > >> > > I was asking because we were trying to reuse some of the
> > > > >> implementations
> > > > >> > in
> > > > >> > > the file source for Iceberg source. Flink Iceberg source lives
> > in
> > > > the
> > > > >> > > Iceberg repo, which is not possible to code against the master
> > > > branch
> > > > >> of
> > > > >> > > the Flink code.
> > > > >> > >
> > > > >> > > On Mon, Nov 2, 2020 at 3:31 AM Stephan Ewen <sewen@apache.org
> >
> > > > wrote:
> > > > >> > >
> > > > >> > > > Hi Steven!
> > > > >> > > >
> > > > >> > > > So far there are no plans to pick back the file system
> > connector
> > > > >> code.
> > > > >> > > This
> > > > >> > > > is still evolving and not finalized for 1.12, so I don't
> feel
> > it
> > > > is
> > > > >> a
> > > > >> > > good
> > > > >> > > > candidate to be backported.
> > > > >> > > > However, with the base connector changes backported, you
> > should
> > > be
> > > > >> able
> > > > >> > > to
> > > > >> > > > run the file connector code from master against 1.11.3.
> > > > >> > > >
> > > > >> > > > The collect() utils can be picked back, I see no issue with
> > that
> > > > >> (it is
> > > > >> > > > isolated utilities).
> > > > >> > > >
> > > > >> > > > Best,
> > > > >> > > > Stephan
> > > > >> > > >
> > > > >> > > >
> > > > >> > > > On Mon, Nov 2, 2020 at 3:02 AM Steven Wu <
> > stevenz3wu@gmail.com>
> > > > >> wrote:
> > > > >> > > >
> > > > >> > > > > Basically, it would be great to get the latest code in the
> > > > >> > > > > flink-connector-files (FLIP-27).
> > > > >> > > > >
> > > > >> > > > > On Sat, Oct 31, 2020 at 9:57 AM Steven Wu <
> > > stevenz3wu@gmail.com
> > > > >
> > > > >> > > wrote:
> > > > >> > > > >
> > > > >> > > > > > Stephan, it will be great if we can also backport the
> > > > >> > DataStreamUtils
> > > > >> > > > > > related commits that help with collecting output from
> > > > unbounded
> > > > >> > > > streams.
> > > > >> > > > > > e.g.
> > > > >> > > > > >
> > > > >> > > > > >
> > > > >> > > > > >
> > > > >> > > > >
> > > > >> > > >
> > > > >> > >
> > > > >> >
> > > > >>
> > > >
> > >
> >
> https://github.com/apache/flink/commit/09a7a66b7313fea64817fe960a8da1265b428efc
> > > > >> > > > > >
> > > > >> > > > > > I tried to copy and paste the code to unblock myself.
> but
> > it
> > > > >> > quickly
> > > > >> > > > got
> > > > >> > > > > > into the rabbit hole of more and more code.
> > > > >> > > > > >
> > > > >> > > > > > On Fri, Oct 30, 2020 at 11:02 AM Stephan Ewen <
> > > > sewen@apache.org
> > > > >> >
> > > > >> > > > wrote:
> > > > >> > > > > >
> > > > >> > > > > >> I have started with backporting the source API changes.
> > > Some
> > > > >> minor
> > > > >> > > > > >> conflicts to solve, will need a bit more to finish
> this.
> > > > >> > > > > >>
> > > > >> > > > > >> On Fri, Oct 30, 2020 at 7:25 AM Tzu-Li (Gordon) Tai <
> > > > >> > > > > tzulitai@apache.org>
> > > > >> > > > > >> wrote:
> > > > >> > > > > >>
> > > > >> > > > > >> > @Stephan Ewen <se...@apache.org>
> > > > >> > > > > >> > Are there already plans or ongoing efforts for
> > > backporting
> > > > >> the
> > > > >> > > list
> > > > >> > > > of
> > > > >> > > > > >> > FLIP-27 changes that you posted?
> > > > >> > > > > >> >
> > > > >> > > > > >> > On Thu, Oct 29, 2020 at 7:08 PM Xintong Song <
> > > > >> > > tonysong820@gmail.com
> > > > >> > > > >
> > > > >> > > > > >> > wrote:
> > > > >> > > > > >> >
> > > > >> > > > > >> >> Hi folks,
> > > > >> > > > > >> >>
> > > > >> > > > > >> >> Just to provide some updates concerning the status
> on
> > > the
> > > > >> > > > > >> >> test instabilities.
> > > > >> > > > > >> >>
> > > > >> > > > > >> >> Currently, we have 30 unresolved tickets labeled
> with
> > > > >> `Affects
> > > > >> > > > > Version`
> > > > >> > > > > >> >> 1.11.x.
> > > > >> > > > > >> >>
> > > > >> > > > > >> >>
> > > > >> > > > > >>
> > > > >> > > > >
> > > > >> > > >
> > > > >> > >
> > > > >> >
> > > > >>
> > > >
> > >
> >
> https://issues.apache.org/jira/browse/FLINK-19775?filter=12348580&jql=project%20%3D%20FLINK%20AND%20resolution%20%3D%20Unresolved%20AND%20affectedVersion%20in%20(1.11.0%2C%201.11.1%2C%201.11.2%2C%201.11.3)%20AND%20labels%20%3D%20test-stability%20ORDER%20BY%20created%20DESC
> > > > >> > > > > >> >>
> > > > >> > > > > >> >> Among the 30 tickets, 11 of them are:
> > > > >> > > > > >> >> - Have occured in the recent 3 months
> > > > >> > > > > >> >> - Not confirmed to be pure testability issues
> > > > >> > > > > >> >> - Not confirmed to be rare condition cases
> > > > >> > > > > >> >>
> > > > >> > > > > >> >> It would be nice if someone familiar with these
> > > components
> > > > >> can
> > > > >> > > > take a
> > > > >> > > > > >> look
> > > > >> > > > > >> >> into these issues.
> > > > >> > > > > >> >>
> > > > >> > > > > >> >> - https://issues.apache.org/jira/browse/FLINK-17159
> > > (ES6)
> > > > >> > > > > >> >> - https://issues.apache.org/jira/browse/FLINK-17912
> > > > (Kafka)
> > > > >> > > > > >> >> - https://issues.apache.org/jira/browse/FLINK-17949
> > > > (Kafka)
> > > > >> > > > > >> >> ⁃ https://issues.apache.org/jira/browse/FLINK-18444
> > > > (Kafka)
> > > > >> > > > > >> >> - https://issues.apache.org/jira/browse/FLINK-18634
> > > > (Kafka)
> > > > >> > > > > >> >> - https://issues.apache.org/jira/browse/FLINK-18648
> > > > (Kafka)
> > > > >> > > > > >> >> - https://issues.apache.org/jira/browse/FLINK-18807
> > > > (Kafka)
> > > > >> > > > > >> >> - https://issues.apache.org/jira/browse/FLINK-19369
> > > > >> > > > (BlobClientTest)
> > > > >> > > > > >> >> - https://issues.apache.org/jira/browse/FLINK-19436
> > > > (TPCDS)
> > > > >> > > > > >> >> - https://issues.apache.org/jira/browse/FLINK-19690
> > > > >> > > > (Format/Parquet)
> > > > >> > > > > >> >> - https://issues.apache.org/jira/browse/FLINK-19775
> > > > >> > > > > >> >> (SystemProcessingTimeServiceTest)
> > > > >> > > > > >> >>
> > > > >> > > > > >> >> Thank you~
> > > > >> > > > > >> >>
> > > > >> > > > > >> >> Xintong Song
> > > > >> > > > > >> >>
> > > > >> > > > > >> >>
> > > > >> > > > > >> >>
> > > > >> > > > > >> >> On Thu, Oct 29, 2020 at 10:21 AM Jingsong Li <
> > > > >> > > > jingsonglee0@gmail.com
> > > > >> > > > > >
> > > > >> > > > > >> >> wrote:
> > > > >> > > > > >> >>
> > > > >> > > > > >> >> > +1 to backport the FLIP-27 adjustments to 1.11.x.
> > > > >> > > > > >> >> >
> > > > >> > > > > >> >> > If possible, that would be great. Many people are
> > > > looking
> > > > >> > > forward
> > > > >> > > > > to
> > > > >> > > > > >> the
> > > > >> > > > > >> >> > FLIP-27 interface, but they don't want to take the
> > > risk
> > > > to
> > > > >> > > > upgrade
> > > > >> > > > > to
> > > > >> > > > > >> >> 1.12
> > > > >> > > > > >> >> > (And wait 1.12). After all, 1.11 is a relatively
> > > stable
> > > > >> > > version.
> > > > >> > > > > >> >> >
> > > > >> > > > > >> >> > Best,
> > > > >> > > > > >> >> > Jingsong
> > > > >> > > > > >> >> >
> > > > >> > > > > >> >> > On Thu, Oct 29, 2020 at 1:24 AM Stephan Ewen <
> > > > >> > sewen@apache.org
> > > > >> > > >
> > > > >> > > > > >> wrote:
> > > > >> > > > > >> >> >
> > > > >> > > > > >> >> > > Thanks for starting this.
> > > > >> > > > > >> >> > >
> > > > >> > > > > >> >> > > +1 form my side to backport the FLIP-27
> > adjustments
> > > to
> > > > >> > > 1.11.x.
> > > > >> > > > > >> >> > >
> > > > >> > > > > >> >> > > There were quite a few changes, and I think we
> > need
> > > to
> > > > >> > > > > cherry-pick
> > > > >> > > > > >> >> them
> > > > >> > > > > >> >> > all
> > > > >> > > > > >> >> > > to not get some inconsistent mix of changes and
> > many
> > > > >> merge
> > > > >> > > > > >> conflicts.
> > > > >> > > > > >> >> > > I made a list below of what we need to add to
> > > > >> > "release-1.11".
> > > > >> > > > > >> >> > >
> > > > >> > > > > >> >> > > *#### Core Source API Changes to backport (in
> > > REVERSE
> > > > >> > order)*
> > > > >> > > > > >> >> > >
> > > > >> > > > > >> >> > >   (Use: "git log
> > > > >> > > > > >> >> > >
> > > > >> > > >
> > flink-core/src/main/java/org/apache/flink/api/connector/source")
> > > > >> > > > > >> >> > >
> > > > >> > > > > >> >> > > commit 162c072e9265a7b6dd9d6f5459eb7974091c4c4e
> > > > >> > > > > >> >> > > [FLINK-19492][core] Consolidate Source Events
> > > between
> > > > >> > Source
> > > > >> > > > API
> > > > >> > > > > >> and
> > > > >> > > > > >> >> > Split
> > > > >> > > > > >> >> > > Reader API
> > > > >> > > > > >> >> > >
> > > > >> > > > > >> >> > > commit 428fbbad85379f71a9fbf1a1ab3e3dcfa9006436
> > > > >> > > > > >> >> > > [FLINK-19205][core] Add access to configuration
> > and
> > > > >> > hostname
> > > > >> > > in
> > > > >> > > > > the
> > > > >> > > > > >> >> > > SourceReaderContext
> > > > >> > > > > >> >> > >
> > > > >> > > > > >> >> > > commit 723e1790744ddc78d19c7c978442af1383f38d33
> > > > >> > > > > >> >> > > [hotfix][core] Add to Source Enumerator
> > convenience
> > > > >> methods
> > > > >> > > to
> > > > >> > > > > >> assign
> > > > >> > > > > >> >> > > single split
> > > > >> > > > > >> >> > >
> > > > >> > > > > >> >> > > *#### Connector Base Changes to Backport (in
> > REVERSE
> > > > >> > order)*
> > > > >> > > > > >> >> > >
> > > > >> > > > > >> >> > >   (Use: "git log
> > > > flink-connectors/flink-connector-base")
> > > > >> > > > > >> >> > >
> > > > >> > > > > >> >> > > commit 401f56fe9d6b0271260edf9787cdcbfe4d03874d
> > > > >> > > > > >> >> > > [FLINK-19427][FLINK-19489][tests] Fix test
> > > conditions
> > > > >> for
> > > > >> > > > > >> >> > >
> > > > 'SplitFetcherTest.testNotifiesWhenGoingIdleConcurrent()'
> > > > >> > > > > >> >> > >
> > > > >> > > > > >> >> > > commit 68c5c2ff779d82a1ff81ffaf60d8a1b283797db1
> > > > >> > > > > >> >> > > [FLINK-19448][connector base] Explicitly check
> for
> > > > >> > > un-expected
> > > > >> > > > > >> >> condition
> > > > >> > > > > >> >> > > that would leave an inconsistent state
> > > > >> > > > > >> >> > >
> > > > >> > > > > >> >> > > commit 162c072e9265a7b6dd9d6f5459eb7974091c4c4e
> > > > >> > > > > >> >> > > [FLINK-19492][core] Consolidate Source Events
> > > between
> > > > >> > Source
> > > > >> > > > API
> > > > >> > > > > >> and
> > > > >> > > > > >> >> > Split
> > > > >> > > > > >> >> > > Reader API
> > > > >> > > > > >> >> > >
> > > > >> > > > > >> >> > > commit c1ca7a4c7c21ec8868c14cdddd43c559625b794c
> > > > >> > > > > >> >> > > [refactor][tests] Move some source test utils
> from
> > > > >> > > > > >> >> flink-connector-base
> > > > >> > > > > >> >> > to
> > > > >> > > > > >> >> > > flink-core
> > > > >> > > > > >> >> > >
> > > > >> > > > > >> >> > > commit ee5c4c211c35c70d28252363bbc8400453609977
> > > > >> > > > > >> >> > > [FLINK-19251][connectors] Avoid confusing queue
> > > > >> handling in
> > > > >> > > > > >> >> > > "SplitReader.handleSplitsChanges()"
> > > > >> > > > > >> >> > >
> > > > >> > > > > >> >> > > commit 5abef56b2bf85bcac786f6b16b6899b6cced7176
> > > > >> > > > > >> >> > > [FLINK-19250][connectors] Fix error propagation
> in
> > > > >> > connector
> > > > >> > > > base
> > > > >> > > > > >> >> > > (SplitFetcherManager).
> > > > >> > > > > >> >> > >
> > > > >> > > > > >> >> > > commit 8fcca837c55a9216595ee4c03038b52747098dbb
> > > > >> > > > > >> >> > > [hotfix][connectors] Improve JavaDocs for
> > > > >> > > > > >> SingleThreadFetcherManager
> > > > >> > > > > >> >> > >
> > > > >> > > > > >> >> > > commit 4700bb5dde3303cbe98882f6beb7379425717b01
> > > > >> > > > > >> >> > > [FLINK-19225][connectors] Various small
> > improvements
> > > > to
> > > > >> > > > > >> >> SourceReaderBase
> > > > >> > > > > >> >> > > (part 2)
> > > > >> > > > > >> >> > >
> > > > >> > > > > >> >> > > commit 12261c6b7ed6478a9b9f6a69cb58246b83cab9b7
> > > > >> > > > > >> >> > > [FLINK-17393][connectors] (follow-up) Wakeup the
> > > > >> > > SplitFetchers
> > > > >> > > > > more
> > > > >> > > > > >> >> > > elegantly.
> > > > >> > > > > >> >> > >
> > > > >> > > > > >> >> > > commit c60aaff0249bfd6b5871b7f82e03efc487a54d6b
> > > > >> > > > > >> >> > > [hotfix][tests] Extend test coverage for
> > > > >> > > > > >> FutureCompletingBlockingQueue
> > > > >> > > > > >> >> > >
> > > > >> > > > > >> >> > > commit cef8a587d7fd2fe64cc644da5ed095d82e46f631
> > > > >> > > > > >> >> > > [FLINK-19245][connectors] Set default capacity
> for
> > > > >> > > > > >> >> > > FutureCompletingBlockingQueue.
> > > > >> > > > > >> >> > >
> > > > >> > > > > >> >> > > commit 4ea95782b4c6a2538153d4d16ad3f4839c7de0fb
> > > > >> > > > > >> >> > > [FLINK-19223][connectors] Simplify Availability
> > > Future
> > > > >> > Model
> > > > >> > > in
> > > > >> > > > > >> Base
> > > > >> > > > > >> >> > > Connector
> > > > >> > > > > >> >> > >
> > > > >> > > > > >> >> > > commit 511857049ba30c8ff0ee56da551fa4a479dc583e
> > > > >> > > > > >> >> > > [FLINK-18128][connectors] Ensure idle split
> > fetchers
> > > > >> lead
> > > > >> > to
> > > > >> > > > > >> >> availability
> > > > >> > > > > >> >> > > notifications.
> > > > >> > > > > >> >> > >
> > > > >> > > > > >> >> > > commit a8206467af0830dcb89623ea068b5ca3b3450c92
> > > > >> > > > > >> >> > > [refactor][core] Eagerly initialize the
> FetchTask
> > to
> > > > >> > support
> > > > >> > > > > proper
> > > > >> > > > > >> >> unit
> > > > >> > > > > >> >> > > testing
> > > > >> > > > > >> >> > >
> > > > >> > > > > >> >> > > commit 3b2f54bcb437f98e6137c904045cc51072b5c06b
> > > > >> > > > > >> >> > > [hotfix][tests] Move constants in
> SplitFetcherTest
> > > > >> relevant
> > > > >> > > to
> > > > >> > > > > only
> > > > >> > > > > >> >> one
> > > > >> > > > > >> >> > > test into test method
> > > > >> > > > > >> >> > >
> > > > >> > > > > >> >> > > commit d7625760a75a508bf05bcddc380bb4d62ee1743e
> > > > >> > > > > >> >> > > [FLINK-19225][connectors] Various small
> > improvements
> > > > to
> > > > >> > > > > >> >> SourceReaderBase
> > > > >> > > > > >> >> > >
> > > > >> > > > > >> >> > > commit a5b0d3297748c1be47ad579a88f24df2255a8df1
> > > > >> > > > > >> >> > > [FLINK-17393][connectors] Wakeup the
> SplitFetchers
> > > > more
> > > > >> > > > > elegantly.
> > > > >> > > > > >> >> > >
> > > > >> > > > > >> >> > > commit f42a3ebc3e81a034b7221a803c153636fef34903
> > > > >> > > > > >> >> > > [FLINK-18680][connectors] Make connector base
> > > > >> > > > RecordsWithSplitIds
> > > > >> > > > > >> more
> > > > >> > > > > >> >> > > lightweight.
> > > > >> > > > > >> >> > >
> > > > >> > > > > >> >> > > commit e3d273de822b085183d09b275a445879ff94b350
> > > > >> > > > > >> >> > > [FLINK-19162][connectors] Add 'recycle()' to the
> > > > >> > > > > >> RecordsWithSplitIds
> > > > >> > > > > >> >> to
> > > > >> > > > > >> >> > > support reuse of heavy objects.
> > > > >> > > > > >> >> > >
> > > > >> > > > > >> >> > > commit 8ebc464c2520453a70001cd712abc8dee6ee89e0
> > > > >> > > > > >> >> > > [hotfix][testing] Add a set of parameterizable
> > > testing
> > > > >> > mocks
> > > > >> > > > for
> > > > >> > > > > >> the
> > > > >> > > > > >> >> > Split
> > > > >> > > > > >> >> > > Reader API
> > > > >> > > > > >> >> > >
> > > > >> > > > > >> >> > > commit 930a07438be1185388d7150640f294dfe2a5d378
> > > > >> > > > > >> >> > > [hotfix][connectors] Add RequestSplitEvent to
> > > > >> > > > > >> 'flink-connector-base'
> > > > >> > > > > >> >> > >
> > > > >> > > > > >> >> > > commit e945ce8a933bc378844782f784ca473c767ca159
> > > > >> > > > > >> >> > > [refactor][connectors] Backport of the
> > > connector-base
> > > > >> > > exception
> > > > >> > > > > >> >> handling
> > > > >> > > > > >> >> > > from the Kafka Connector Pull Request
> > > > >> > > > > >> >> > >
> > > > >> > > > > >> >> > >
> > > > >> > > > > >> >> > > On Wed, Oct 28, 2020 at 5:49 PM Steven Wu <
> > > > >> > > > stevenz3wu@gmail.com>
> > > > >> > > > > >> >> wrote:
> > > > >> > > > > >> >> > >
> > > > >> > > > > >> >> > > > I would love to see this FLIP-27 source
> > interface
> > > > >> > > improvement
> > > > >> > > > > [1]
> > > > >> > > > > >> >> made
> > > > >> > > > > >> >> > to
> > > > >> > > > > >> >> > > > 1.11.3.
> > > > >> > > > > >> >> > > >
> > > > >> > > > > >> >> > > > [1]
> > > > https://issues.apache.org/jira/browse/FLINK-19698
> > > > >> > > > > >> >> > > >
> > > > >> > > > > >> >> > > > On Wed, Oct 28, 2020 at 12:32 AM Tzu-Li
> (Gordon)
> > > > Tai <
> > > > >> > > > > >> >> > > tzulitai@apache.org>
> > > > >> > > > > >> >> > > > wrote:
> > > > >> > > > > >> >> > > >
> > > > >> > > > > >> >> > > > > Thanks for the replies so far!
> > > > >> > > > > >> >> > > > >
> > > > >> > > > > >> >> > > > > Just to provide a brief update on the status
> > of
> > > > >> > blockers
> > > > >> > > > for
> > > > >> > > > > >> >> 1.11.3
> > > > >> > > > > >> >> > so
> > > > >> > > > > >> >> > > > far:
> > > > >> > > > > >> >> > > > >
> > > > >> > > > > >> >> > > > >
> > > > >> > > > > >> >> > > > > *PR opened, pending reviewer*- [FLINK-19717]
> > > > >> > > > > >> >> > SourceReaderBase.pollNext
> > > > >> > > > > >> >> > > > may
> > > > >> > > > > >> >> > > > > return END_OF_INPUT if SplitReader.fetch
> > throws
> > > (
> > > > >> > > > > >> >> > > > > https://github.com/apache/flink/pull/13776)
> > > > >> > > > > >> >> > > > >
> > > > >> > > > > >> >> > > > > *PR opened, reviewed + close to being
> merged*
> > > > >> > > > > >> >> > > > > - [FLINK-19741] Timer service should skip
> > > > restoring
> > > > >> > from
> > > > >> > > > raw
> > > > >> > > > > >> keyed
> > > > >> > > > > >> >> > > stream
> > > > >> > > > > >> >> > > > > if it isn't the writer (
> > > > >> > > > > >> >> https://github.com/apache/flink/pull/13761)
> > > > >> > > > > >> >> > > > > - [FLINK-19748] Raw keyed stream key group
> > > > iterator
> > > > >> > > should
> > > > >> > > > be
> > > > >> > > > > >> >> > skipping
> > > > >> > > > > >> >> > > > > unwritten key groups (
> > > > >> > > > > >> https://github.com/apache/flink/pull/13772)
> > > > >> > > > > >> >> > > > >
> > > > >> > > > > >> >> > > > > *Merged*
> > > > >> > > > > >> >> > > > > - [FLINK-19154] Application mode deletes HA
> > data
> > > > in
> > > > >> > case
> > > > >> > > of
> > > > >> > > > > >> >> suspended
> > > > >> > > > > >> >> > > > > ZooKeeper connection
> > > > >> > > > > >> >> > > > > - [FLINK-19569] Upgrade ICU4J to 67.1+
> > > > >> > > > > >> >> > > > >
> > > > >> > > > > >> >> > > > > Right now as it seems, progress is mainly
> > > blocked
> > > > >> on a
> > > > >> > > > > reviewer
> > > > >> > > > > >> >> for
> > > > >> > > > > >> >> > > > > FLINK-19717.
> > > > >> > > > > >> >> > > > > Meanwhile, Xintong is keeping an eye on test
> > > > >> > > instabilities
> > > > >> > > > > [1]
> > > > >> > > > > >> to
> > > > >> > > > > >> >> see
> > > > >> > > > > >> >> > > if
> > > > >> > > > > >> >> > > > > there are any fixes that should be applied
> to
> > > > >> > > > `release-1.11`.
> > > > >> > > > > >> >> > > > >
> > > > >> > > > > >> >> > > > > This is also a reminder, that if there are
> > other
> > > > >> > blockers
> > > > >> > > > > that
> > > > >> > > > > >> we
> > > > >> > > > > >> >> > need
> > > > >> > > > > >> >> > > to
> > > > >> > > > > >> >> > > > > be aware of, or a need to re-establish
> > estimated
> > > > >> time
> > > > >> > for
> > > > >> > > > > >> getting
> > > > >> > > > > >> >> > fixes
> > > > >> > > > > >> >> > > > in
> > > > >> > > > > >> >> > > > > and delay the RC for 1.11.3, please do let
> us
> > > > know!
> > > > >> > > > > >> >> > > > >
> > > > >> > > > > >> >> > > > > Cheers,
> > > > >> > > > > >> >> > > > > Gordon
> > > > >> > > > > >> >> > > > >
> > > > >> > > > > >> >> > > > > [1]
> > > > >> > > > > >> >> > > > >
> > > > >> > > > > >> >> > > > >
> > > > >> > > > > >> >> > > >
> > > > >> > > > > >> >> > >
> > > > >> > > > > >> >> >
> > > > >> > > > > >> >>
> > > > >> > > > > >>
> > > > >> > > > >
> > > > >> > > >
> > > > >> > >
> > > > >> >
> > > > >>
> > > >
> > >
> >
> https://issues.apache.org/jira/issues/?jql=project%20%3D%20FLINK%20AND%20status%20in%20(Open%2C%20%22In%20Progress%22%2C%20Reopened)%20AND%20affectedVersion%20in%20(1.11.0%2C%201.11.1%2C%201.11.2)%20AND%20labels%20%3D%20test-stability
> > > > >> > > > > >> >> > > > >
> > > > >> > > > > >> >> > > > > On Mon, Oct 26, 2020 at 9:43 PM Kostas
> > Kloudas <
> > > > >> > > > > >> >> kkloudas@apache.org>
> > > > >> > > > > >> >> > > > > wrote:
> > > > >> > > > > >> >> > > > >
> > > > >> > > > > >> >> > > > > > +1 for releasing Flink 1.11.3 as it
> > contains a
> > > > >> number
> > > > >> > > of
> > > > >> > > > > >> >> important
> > > > >> > > > > >> >> > > > > > fixes and thanks Gordon and Xintong for
> > > > >> volunteering.
> > > > >> > > > > >> >> > > > > >
> > > > >> > > > > >> >> > > > > > Cheers,
> > > > >> > > > > >> >> > > > > > Kostas
> > > > >> > > > > >> >> > > > > >
> > > > >> > > > > >> >> > > > > > On Mon, Oct 26, 2020 at 4:37 AM Yu Li <
> > > > >> > > carp84@gmail.com>
> > > > >> > > > > >> wrote:
> > > > >> > > > > >> >> > > > > > >
> > > > >> > > > > >> >> > > > > > > +1 for releasing Flink 1.11.3, and
> thanks
> > > > Gordon
> > > > >> > and
> > > > >> > > > > >> Xintong
> > > > >> > > > > >> >> for
> > > > >> > > > > >> >> > > > > > > volunteering as our release managers.
> > > > >> > > > > >> >> > > > > > >
> > > > >> > > > > >> >> > > > > > > Best Regards,
> > > > >> > > > > >> >> > > > > > > Yu
> > > > >> > > > > >> >> > > > > > >
> > > > >> > > > > >> >> > > > > > >
> > > > >> > > > > >> >> > > > > > > On Mon, 26 Oct 2020 at 09:45, Xintong
> > Song <
> > > > >> > > > > >> >> > tonysong820@gmail.com>
> > > > >> > > > > >> >> > > > > > wrote:
> > > > >> > > > > >> >> > > > > > >
> > > > >> > > > > >> >> > > > > > > > Thanks Gordan for starting this
> > > discussion.
> > > > >> > > > > >> >> > > > > > > > My pleasure to help with the release
> > > > process.
> > > > >> > > > > >> >> > > > > > > >
> > > > >> > > > > >> >> > > > > > > > Thank you~
> > > > >> > > > > >> >> > > > > > > >
> > > > >> > > > > >> >> > > > > > > > Xintong Song
> > > > >> > > > > >> >> > > > > > > >
> > > > >> > > > > >> >> > > > > > > >
> > > > >> > > > > >> >> > > > > > > >
> > > > >> > > > > >> >> > > > > > > > On Fri, Oct 23, 2020 at 11:29 PM Till
> > > > >> Rohrmann <
> > > > >> > > > > >> >> > > > trohrmann@apache.org
> > > > >> > > > > >> >> > > > > >
> > > > >> > > > > >> >> > > > > > > > wrote:
> > > > >> > > > > >> >> > > > > > > >
> > > > >> > > > > >> >> > > > > > > > > Thanks for starting this discussion
> > > > Gordon.
> > > > >> > There
> > > > >> > > > are
> > > > >> > > > > >> over
> > > > >> > > > > >> >> > 100
> > > > >> > > > > >> >> > > > > issues
> > > > >> > > > > >> >> > > > > > > > > which are fixed for 1.11.3. Hence +1
> > > for a
> > > > >> > > soonish
> > > > >> > > > > >> 1.11.3
> > > > >> > > > > >> >> > > > release.
> > > > >> > > > > >> >> > > > > > Thanks
> > > > >> > > > > >> >> > > > > > > > > for volunteering as our release
> > managers
> > > > >> Gordon
> > > > >> > > and
> > > > >> > > > > >> >> Xintong!
> > > > >> > > > > >> >> > > > > > > > >
> > > > >> > > > > >> >> > > > > > > > > Cheers,
> > > > >> > > > > >> >> > > > > > > > > Till
> > > > >> > > > > >> >> > > > > > > > >
> > > > >> > > > > >> >> > > > > > > > > On Fri, Oct 23, 2020 at 5:02 PM
> Tzu-Li
> > > > >> (Gordon)
> > > > >> > > > Tai <
> > > > >> > > > > >> >> > > > > > tzulitai@apache.org
> > > > >> > > > > >> >> > > > > > > > >
> > > > >> > > > > >> >> > > > > > > > > wrote:
> > > > >> > > > > >> >> > > > > > > > >
> > > > >> > > > > >> >> > > > > > > > >> Hi,
> > > > >> > > > > >> >> > > > > > > > >>
> > > > >> > > > > >> >> > > > > > > > >> Xintong and I would like to start a
> > > > >> discussion
> > > > >> > > for
> > > > >> > > > > >> >> releasing
> > > > >> > > > > >> >> > > > Flink
> > > > >> > > > > >> >> > > > > > > > 1.11.3
> > > > >> > > > > >> >> > > > > > > > >> soon.
> > > > >> > > > > >> >> > > > > > > > >>
> > > > >> > > > > >> >> > > > > > > > >> It seems like we already have a few
> > > > >> pressing
> > > > >> > > > issues
> > > > >> > > > > >> that
> > > > >> > > > > >> >> > needs
> > > > >> > > > > >> >> > > > to
> > > > >> > > > > >> >> > > > > be
> > > > >> > > > > >> >> > > > > > > > >> included in a new hotfix release:
> > > > >> > > > > >> >> > > > > > > > >>
> > > > >> > > > > >> >> > > > > > > > >>    - Heap-based timers’ restore
> > > behaviour
> > > > >> is
> > > > >> > > > > causing a
> > > > >> > > > > >> >> > > critical
> > > > >> > > > > >> >> > > > > > recovery
> > > > >> > > > > >> >> > > > > > > > >>    issue for StateFun [1] [2] [3].
> > > > >> > > > > >> >> > > > > > > > >>    - There are several robustness
> > > issues
> > > > >> for
> > > > >> > the
> > > > >> > > > > >> FLIP-27
> > > > >> > > > > >> >> new
> > > > >> > > > > >> >> > > > > source
> > > > >> > > > > >> >> > > > > > API,
> > > > >> > > > > >> >> > > > > > > > >>    such as [4]. We already have
> some
> > > > users
> > > > >> > using
> > > > >> > > > the
> > > > >> > > > > >> >> FLIP-27
> > > > >> > > > > >> >> > > API
> > > > >> > > > > >> >> > > > > > with
> > > > >> > > > > >> >> > > > > > > > >> 1.11.x,
> > > > >> > > > > >> >> > > > > > > > >>    so it would be important to get
> > > those
> > > > >> fixes
> > > > >> > > in
> > > > >> > > > > for
> > > > >> > > > > >> >> 1.11.x
> > > > >> > > > > >> >> > > as
> > > > >> > > > > >> >> > > > > > well.
> > > > >> > > > > >> >> > > > > > > > >>
> > > > >> > > > > >> >> > > > > > > > >> Apart from the issues that are
> > already
> > > > >> marked
> > > > >> > as
> > > > >> > > > > >> blocker
> > > > >> > > > > >> >> for
> > > > >> > > > > >> >> > > > > 1.11.3
> > > > >> > > > > >> >> > > > > > in
> > > > >> > > > > >> >> > > > > > > > our
> > > > >> > > > > >> >> > > > > > > > >> JIRA [5], please let us know in
> this
> > > > >> thread if
> > > > >> > > > there
> > > > >> > > > > >> is
> > > > >> > > > > >> >> > > already
> > > > >> > > > > >> >> > > > > > ongoing
> > > > >> > > > > >> >> > > > > > > > >> work for other important fixes that
> > we
> > > > >> should
> > > > >> > > try
> > > > >> > > > to
> > > > >> > > > > >> >> > include.
> > > > >> > > > > >> >> > > > > > > > >>
> > > > >> > > > > >> >> > > > > > > > >> Xintong and I would like to
> volunteer
> > > for
> > > > >> > > managing
> > > > >> > > > > >> this
> > > > >> > > > > >> >> > > release,
> > > > >> > > > > >> >> > > > > and
> > > > >> > > > > >> >> > > > > > > > will
> > > > >> > > > > >> >> > > > > > > > >> try to communicate the priority of
> > > > pending
> > > > >> > > > blockers
> > > > >> > > > > >> over
> > > > >> > > > > >> >> the
> > > > >> > > > > >> >> > > > next
> > > > >> > > > > >> >> > > > > > few
> > > > >> > > > > >> >> > > > > > > > >> days.
> > > > >> > > > > >> >> > > > > > > > >> Since the aforementioned issues are
> > > quite
> > > > >> > > > critical,
> > > > >> > > > > >> we’d
> > > > >> > > > > >> >> > like
> > > > >> > > > > >> >> > > to
> > > > >> > > > > >> >> > > > > aim
> > > > >> > > > > >> >> > > > > > > > >> for a *feature
> > > > >> > > > > >> >> > > > > > > > >> freeze by the end of next week
> (Oct.
> > > > 30th)*
> > > > >> > and
> > > > >> > > > > start
> > > > >> > > > > >> the
> > > > >> > > > > >> >> > > > release
> > > > >> > > > > >> >> > > > > > voting
> > > > >> > > > > >> >> > > > > > > > >> process the week after.
> > > > >> > > > > >> >> > > > > > > > >> If that is too short of a notice
> and
> > > you
> > > > >> might
> > > > >> > > > need
> > > > >> > > > > >> more
> > > > >> > > > > >> >> > time,
> > > > >> > > > > >> >> > > > > > please
> > > > >> > > > > >> >> > > > > > > > let
> > > > >> > > > > >> >> > > > > > > > >> us know!
> > > > >> > > > > >> >> > > > > > > > >>
> > > > >> > > > > >> >> > > > > > > > >> Cheers,
> > > > >> > > > > >> >> > > > > > > > >> Gordon
> > > > >> > > > > >> >> > > > > > > > >>
> > > > >> > > > > >> >> > > > > > > > >> [1]
> > > > >> > > > > https://issues.apache.org/jira/browse/FLINK-19692
> > > > >> > > > > >> >> > > > > > > > >> [2]
> > > > >> > > > > https://issues.apache.org/jira/browse/FLINK-19741
> > > > >> > > > > >> >> > > > > > > > >> [3]
> > > > >> > > > > https://issues.apache.org/jira/browse/FLINK-19748
> > > > >> > > > > >> >> > > > > > > > >> [4]
> > > > >> > > > > https://issues.apache.org/jira/browse/FLINK-19717
> > > > >> > > > > >> >> > > > > > > > >> [5]
> > > > >> > > > > >> >> > > > > > > > >>
> > > > >> > > > > >> >> > > > > > > > >>
> > > > >> > > > > >> >> > > > > > > >
> > > > >> > > > > >> >> > > > > >
> > > > >> > > > > >> >> > > > >
> > > > >> > > > > >> >> > > >
> > > > >> > > > > >> >> > >
> > > > >> > > > > >> >> >
> > > > >> > > > > >> >>
> > > > >> > > > > >>
> > > > >> > > > >
> > > > >> > > >
> > > > >> > >
> > > > >> >
> > > > >>
> > > >
> > >
> >
> https://issues.apache.org/jira/issues/?jql=project%20%3D%20FLINK%20AND%20priority%20%3D%20Blocker%20AND%20fixVersion%20%3D%201.11.3
> > > > >> > > > > >> >> > > > > > > > >>
> > > > >> > > > > >> >> > > > > > > > >
> > > > >> > > > > >> >> > > > > > > >
> > > > >> > > > > >> >> > > > > >
> > > > >> > > > > >> >> > > > >
> > > > >> > > > > >> >> > > >
> > > > >> > > > > >> >> > >
> > > > >> > > > > >> >> >
> > > > >> > > > > >> >> >
> > > > >> > > > > >> >> > --
> > > > >> > > > > >> >> > Best, Jingsong Lee
> > > > >> > > > > >> >> >
> > > > >> > > > > >> >>
> > > > >> > > > > >> >
> > > > >> > > > > >>
> > > > >> > > > > >
> > > > >> > > > >
> > > > >> > > >
> > > > >> > >
> > > > >> >
> > > > >>
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] Releasing Apache Flink 1.11.3

Posted by Xintong Song <to...@gmail.com>.
Thanks for the notice, Till.

I just checked and found FLINK-20033 is already fixed. Shall we also
downgrade FLINK-19816 to `Critical`?

Thank you~

Xintong Song



On Mon, Nov 9, 2020 at 4:42 PM Till Rohrmann <tr...@apache.org> wrote:

> I would like to bring one more critical issue to your attention which is
> FLINK-20033 [1]. I believe that this issue is actually causing what has
> been reported in FLINK-19816 [2]. I hope to have it fixed by the end of
> today. Once FLINK-20033 is fixed, I think that we don't have to block the
> release on FLINK-19816.
>
> [1] https://issues.apache.org/jira/browse/FLINK-20033
> [2] https://issues.apache.org/jira/browse/FLINK-19816
>
> Cheers,
> Till
>
> On Mon, Nov 9, 2020 at 4:05 AM Xintong Song <to...@gmail.com> wrote:
>
> > Hi devs,
> >
> > I'd like to provide an update on the progress of preparing release
> 1.11.3.
> >
> > *Blockers*
> > We currently have 3 remaining blockers. (3 resolved and 1 emerged
> compared
> > to last week)
> >
> > - [FLINK-19698] Add close() method and onCheckpointComplete() to the
> > Source.
> > The issue has been fixed on the master branch. It's currently blocked on
> > the FLIP-27 backportings to backport it to the 1.11 branch.
> >
> > - [FLINK-19717] SourceReaderBase.pollNext may return END_OF_INPUT if
> > SplitReader.fetch throws
> > A PR has been opened and reviewed. From the discussions on the PR, it
> looks
> > close to mergeable.
> >
> > - [FLINK-19816] Flink restored from a wrong checkpoint (a very old one
> and
> > not the last completed one)
> > This is a newly emerged blocker and Matthias is working on it.
> >
> > *Test Instabilities*
> > We currently have 27 test instabilities[1].
> > AFAIK, none of them are as serious as to block the 1.11.3 release.
> >
> > *FLIP-27 Backprotings*
> >
> > I noticed that there's no jira issues opened on the FLIP-27 backporting
> > efforts, which is part of the major efforts planned for the 1.11.3
> release,
> > making it hard to track the progress.
> >
> >
> > @Stephan and @Becket, could you please share the updates on the
> backporting
> > efforts? How is the progress and when are the efforts expected to be
> > finished? It would be appreciated and helpful if we can have a jira
> ticket
> > to track the backporing progress publicly.
> >
> >
> > Thank you~
> >
> > Xintong Song
> >
> >
> > [1]
> >
> >
> https://issues.apache.org/jira/browse/FLINK-16908?filter=12348580&jql=project%20%3D%20FLINK%20AND%20resolution%20%3D%20Unresolved%20AND%20affectedVersion%20in%20(1.11.0%2C%201.11.1%2C%201.11.2%2C%201.11.3)%20AND%20labels%20%3D%20test-stability%20ORDER%20BY%20updated%20DESC%2C%20created%20DESC
> >
> > On Wed, Nov 4, 2020 at 3:58 PM Tzu-Li (Gordon) Tai <tz...@apache.org>
> > wrote:
> >
> > > Thanks for the efforts so far with pushing for 1.11.3.
> > >
> > > I'd like to provide a mid-week update on how we're looking with all the
> > > pending blockers and backports:
> > >
> > > *Blockers**:*
> > >
> > > Currently all blockers either have an open PR, or have been merged.
> I'll
> > > highlight below the blockers *that still require reviewing efforts for
> > them
> > > to move forward:*
> > >
> > > - [FLINK-19909] Flink application in attach mode could not terminate
> when
> > > the only job is canceled. PR:
> https://github.com/apache/flink/pull/13911
> > > - [FLINK-19717] SourceReaderBase.pollNext may return END_OF_INPUT if
> > > SplitReader.fetch throws. PR:
> https://github.com/apache/flink/pull/13776
> > >
> > > The above PRs currently have no reviews at all yet. They seem to
> already
> > > have designated reviewers.
> > >
> > >
> > > *Backports:*
> > > - FLIP-27 robustness improvement backports: Stephan is currently
> working
> > on
> > > backporting several FLIP-27 changes. There are no PRs yet for the
> > > backported changes.
> > > - DataStreamUtils.collect() refactorings backport: Steven Wu mentioned
> to
> > > backport this, but AFAIK this isn't assigned to anyone yet.
> > >
> > > *ETAs*:
> > >
> > > I'd like to request ETAs for the remaining backports, to prevent a
> creep
> > in
> > > the scope of this bugfix release.
> > >
> > > We already have Flink users that would benefit from fixes that have
> been
> > > merged to release-1.11 already, so technically speaking the backports
> > > should be considered "nice-to-have" (to the best of my knowledge of the
> > > changes) and could potentially be moved to a follow-up 1.11.4.
> > > Most notably, the Stateful Functions project is already waiting on
> Flink
> > > 1.11.3 to address critical recovery issues (please see the StateFun
> 2.2.1
> > > release discussion thread [1]).
> > >
> > > @Stephan Ewen <se...@apache.org> @Becket Qin <be...@gmail.com>
> > could
> > > you provide an ETA for the FLIP-27 backports? It would help to get a
> > better
> > > estimate to decide how we proceed here.
> > >
> > > Cheers,
> > > Gordon
> > >
> > > [1]
> > >
> > >
> >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Releasing-StateFun-hotfix-version-2-2-1-td46239.html
> > >
> > > On Wed, Nov 4, 2020 at 3:16 PM Tzu-Li (Gordon) Tai <
> tzulitai@apache.org>
> > > wrote:
> > >
> > > > > The collect() utils can be picked back, I see no issue with that
> (it
> > is
> > > > isolated utilities).
> > > >
> > > > Just checking on all the requested backports mentioned in this
> thread,
> > > and
> > > > figuring out which ones seem to still be unassigned / open.
> > > >
> > > > Is someone working on backporting
> > > >
> > >
> >
> https://github.com/apache/flink/commit/09a7a66b7313fea64817fe960a8da1265b428efc
> > > > to release-1.11 at the moment?
> > > >
> > > > On Wed, Nov 4, 2020 at 1:39 AM Steven Wu <st...@gmail.com>
> wrote:
> > > >
> > > >> @Stephan Ewen <se...@apache.org> yeah, we can do that. don't worry
> > > about
> > > >> it. your earlier email had the perfect explanation on why file
> source
> > > >> shouldn't be backported.
> > > >>
> > > >> On Tue, Nov 3, 2020 at 3:37 AM Stephan Ewen <se...@apache.org>
> wrote:
> > > >>
> > > >> > @Steven would it be possible to initially copy some of the code
> into
> > > the
> > > >> > iceberg source and later replace it by a dependency on the Flink
> > file
> > > >> > source?
> > > >> >
> > > >> > On Mon, Nov 2, 2020 at 8:33 PM Steven Wu <st...@gmail.com>
> > > wrote:
> > > >> >
> > > >> > > Stephan, thanks a lot for explaining the file connector. that
> > makes
> > > >> > sense.
> > > >> > >
> > > >> > > I was asking because we were trying to reuse some of the
> > > >> implementations
> > > >> > in
> > > >> > > the file source for Iceberg source. Flink Iceberg source lives
> in
> > > the
> > > >> > > Iceberg repo, which is not possible to code against the master
> > > branch
> > > >> of
> > > >> > > the Flink code.
> > > >> > >
> > > >> > > On Mon, Nov 2, 2020 at 3:31 AM Stephan Ewen <se...@apache.org>
> > > wrote:
> > > >> > >
> > > >> > > > Hi Steven!
> > > >> > > >
> > > >> > > > So far there are no plans to pick back the file system
> connector
> > > >> code.
> > > >> > > This
> > > >> > > > is still evolving and not finalized for 1.12, so I don't feel
> it
> > > is
> > > >> a
> > > >> > > good
> > > >> > > > candidate to be backported.
> > > >> > > > However, with the base connector changes backported, you
> should
> > be
> > > >> able
> > > >> > > to
> > > >> > > > run the file connector code from master against 1.11.3.
> > > >> > > >
> > > >> > > > The collect() utils can be picked back, I see no issue with
> that
> > > >> (it is
> > > >> > > > isolated utilities).
> > > >> > > >
> > > >> > > > Best,
> > > >> > > > Stephan
> > > >> > > >
> > > >> > > >
> > > >> > > > On Mon, Nov 2, 2020 at 3:02 AM Steven Wu <
> stevenz3wu@gmail.com>
> > > >> wrote:
> > > >> > > >
> > > >> > > > > Basically, it would be great to get the latest code in the
> > > >> > > > > flink-connector-files (FLIP-27).
> > > >> > > > >
> > > >> > > > > On Sat, Oct 31, 2020 at 9:57 AM Steven Wu <
> > stevenz3wu@gmail.com
> > > >
> > > >> > > wrote:
> > > >> > > > >
> > > >> > > > > > Stephan, it will be great if we can also backport the
> > > >> > DataStreamUtils
> > > >> > > > > > related commits that help with collecting output from
> > > unbounded
> > > >> > > > streams.
> > > >> > > > > > e.g.
> > > >> > > > > >
> > > >> > > > > >
> > > >> > > > > >
> > > >> > > > >
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > >
> >
> https://github.com/apache/flink/commit/09a7a66b7313fea64817fe960a8da1265b428efc
> > > >> > > > > >
> > > >> > > > > > I tried to copy and paste the code to unblock myself. but
> it
> > > >> > quickly
> > > >> > > > got
> > > >> > > > > > into the rabbit hole of more and more code.
> > > >> > > > > >
> > > >> > > > > > On Fri, Oct 30, 2020 at 11:02 AM Stephan Ewen <
> > > sewen@apache.org
> > > >> >
> > > >> > > > wrote:
> > > >> > > > > >
> > > >> > > > > >> I have started with backporting the source API changes.
> > Some
> > > >> minor
> > > >> > > > > >> conflicts to solve, will need a bit more to finish this.
> > > >> > > > > >>
> > > >> > > > > >> On Fri, Oct 30, 2020 at 7:25 AM Tzu-Li (Gordon) Tai <
> > > >> > > > > tzulitai@apache.org>
> > > >> > > > > >> wrote:
> > > >> > > > > >>
> > > >> > > > > >> > @Stephan Ewen <se...@apache.org>
> > > >> > > > > >> > Are there already plans or ongoing efforts for
> > backporting
> > > >> the
> > > >> > > list
> > > >> > > > of
> > > >> > > > > >> > FLIP-27 changes that you posted?
> > > >> > > > > >> >
> > > >> > > > > >> > On Thu, Oct 29, 2020 at 7:08 PM Xintong Song <
> > > >> > > tonysong820@gmail.com
> > > >> > > > >
> > > >> > > > > >> > wrote:
> > > >> > > > > >> >
> > > >> > > > > >> >> Hi folks,
> > > >> > > > > >> >>
> > > >> > > > > >> >> Just to provide some updates concerning the status on
> > the
> > > >> > > > > >> >> test instabilities.
> > > >> > > > > >> >>
> > > >> > > > > >> >> Currently, we have 30 unresolved tickets labeled with
> > > >> `Affects
> > > >> > > > > Version`
> > > >> > > > > >> >> 1.11.x.
> > > >> > > > > >> >>
> > > >> > > > > >> >>
> > > >> > > > > >>
> > > >> > > > >
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > >
> >
> https://issues.apache.org/jira/browse/FLINK-19775?filter=12348580&jql=project%20%3D%20FLINK%20AND%20resolution%20%3D%20Unresolved%20AND%20affectedVersion%20in%20(1.11.0%2C%201.11.1%2C%201.11.2%2C%201.11.3)%20AND%20labels%20%3D%20test-stability%20ORDER%20BY%20created%20DESC
> > > >> > > > > >> >>
> > > >> > > > > >> >> Among the 30 tickets, 11 of them are:
> > > >> > > > > >> >> - Have occured in the recent 3 months
> > > >> > > > > >> >> - Not confirmed to be pure testability issues
> > > >> > > > > >> >> - Not confirmed to be rare condition cases
> > > >> > > > > >> >>
> > > >> > > > > >> >> It would be nice if someone familiar with these
> > components
> > > >> can
> > > >> > > > take a
> > > >> > > > > >> look
> > > >> > > > > >> >> into these issues.
> > > >> > > > > >> >>
> > > >> > > > > >> >> - https://issues.apache.org/jira/browse/FLINK-17159
> > (ES6)
> > > >> > > > > >> >> - https://issues.apache.org/jira/browse/FLINK-17912
> > > (Kafka)
> > > >> > > > > >> >> - https://issues.apache.org/jira/browse/FLINK-17949
> > > (Kafka)
> > > >> > > > > >> >> ⁃ https://issues.apache.org/jira/browse/FLINK-18444
> > > (Kafka)
> > > >> > > > > >> >> - https://issues.apache.org/jira/browse/FLINK-18634
> > > (Kafka)
> > > >> > > > > >> >> - https://issues.apache.org/jira/browse/FLINK-18648
> > > (Kafka)
> > > >> > > > > >> >> - https://issues.apache.org/jira/browse/FLINK-18807
> > > (Kafka)
> > > >> > > > > >> >> - https://issues.apache.org/jira/browse/FLINK-19369
> > > >> > > > (BlobClientTest)
> > > >> > > > > >> >> - https://issues.apache.org/jira/browse/FLINK-19436
> > > (TPCDS)
> > > >> > > > > >> >> - https://issues.apache.org/jira/browse/FLINK-19690
> > > >> > > > (Format/Parquet)
> > > >> > > > > >> >> - https://issues.apache.org/jira/browse/FLINK-19775
> > > >> > > > > >> >> (SystemProcessingTimeServiceTest)
> > > >> > > > > >> >>
> > > >> > > > > >> >> Thank you~
> > > >> > > > > >> >>
> > > >> > > > > >> >> Xintong Song
> > > >> > > > > >> >>
> > > >> > > > > >> >>
> > > >> > > > > >> >>
> > > >> > > > > >> >> On Thu, Oct 29, 2020 at 10:21 AM Jingsong Li <
> > > >> > > > jingsonglee0@gmail.com
> > > >> > > > > >
> > > >> > > > > >> >> wrote:
> > > >> > > > > >> >>
> > > >> > > > > >> >> > +1 to backport the FLIP-27 adjustments to 1.11.x.
> > > >> > > > > >> >> >
> > > >> > > > > >> >> > If possible, that would be great. Many people are
> > > looking
> > > >> > > forward
> > > >> > > > > to
> > > >> > > > > >> the
> > > >> > > > > >> >> > FLIP-27 interface, but they don't want to take the
> > risk
> > > to
> > > >> > > > upgrade
> > > >> > > > > to
> > > >> > > > > >> >> 1.12
> > > >> > > > > >> >> > (And wait 1.12). After all, 1.11 is a relatively
> > stable
> > > >> > > version.
> > > >> > > > > >> >> >
> > > >> > > > > >> >> > Best,
> > > >> > > > > >> >> > Jingsong
> > > >> > > > > >> >> >
> > > >> > > > > >> >> > On Thu, Oct 29, 2020 at 1:24 AM Stephan Ewen <
> > > >> > sewen@apache.org
> > > >> > > >
> > > >> > > > > >> wrote:
> > > >> > > > > >> >> >
> > > >> > > > > >> >> > > Thanks for starting this.
> > > >> > > > > >> >> > >
> > > >> > > > > >> >> > > +1 form my side to backport the FLIP-27
> adjustments
> > to
> > > >> > > 1.11.x.
> > > >> > > > > >> >> > >
> > > >> > > > > >> >> > > There were quite a few changes, and I think we
> need
> > to
> > > >> > > > > cherry-pick
> > > >> > > > > >> >> them
> > > >> > > > > >> >> > all
> > > >> > > > > >> >> > > to not get some inconsistent mix of changes and
> many
> > > >> merge
> > > >> > > > > >> conflicts.
> > > >> > > > > >> >> > > I made a list below of what we need to add to
> > > >> > "release-1.11".
> > > >> > > > > >> >> > >
> > > >> > > > > >> >> > > *#### Core Source API Changes to backport (in
> > REVERSE
> > > >> > order)*
> > > >> > > > > >> >> > >
> > > >> > > > > >> >> > >   (Use: "git log
> > > >> > > > > >> >> > >
> > > >> > > >
> flink-core/src/main/java/org/apache/flink/api/connector/source")
> > > >> > > > > >> >> > >
> > > >> > > > > >> >> > > commit 162c072e9265a7b6dd9d6f5459eb7974091c4c4e
> > > >> > > > > >> >> > > [FLINK-19492][core] Consolidate Source Events
> > between
> > > >> > Source
> > > >> > > > API
> > > >> > > > > >> and
> > > >> > > > > >> >> > Split
> > > >> > > > > >> >> > > Reader API
> > > >> > > > > >> >> > >
> > > >> > > > > >> >> > > commit 428fbbad85379f71a9fbf1a1ab3e3dcfa9006436
> > > >> > > > > >> >> > > [FLINK-19205][core] Add access to configuration
> and
> > > >> > hostname
> > > >> > > in
> > > >> > > > > the
> > > >> > > > > >> >> > > SourceReaderContext
> > > >> > > > > >> >> > >
> > > >> > > > > >> >> > > commit 723e1790744ddc78d19c7c978442af1383f38d33
> > > >> > > > > >> >> > > [hotfix][core] Add to Source Enumerator
> convenience
> > > >> methods
> > > >> > > to
> > > >> > > > > >> assign
> > > >> > > > > >> >> > > single split
> > > >> > > > > >> >> > >
> > > >> > > > > >> >> > > *#### Connector Base Changes to Backport (in
> REVERSE
> > > >> > order)*
> > > >> > > > > >> >> > >
> > > >> > > > > >> >> > >   (Use: "git log
> > > flink-connectors/flink-connector-base")
> > > >> > > > > >> >> > >
> > > >> > > > > >> >> > > commit 401f56fe9d6b0271260edf9787cdcbfe4d03874d
> > > >> > > > > >> >> > > [FLINK-19427][FLINK-19489][tests] Fix test
> > conditions
> > > >> for
> > > >> > > > > >> >> > >
> > > 'SplitFetcherTest.testNotifiesWhenGoingIdleConcurrent()'
> > > >> > > > > >> >> > >
> > > >> > > > > >> >> > > commit 68c5c2ff779d82a1ff81ffaf60d8a1b283797db1
> > > >> > > > > >> >> > > [FLINK-19448][connector base] Explicitly check for
> > > >> > > un-expected
> > > >> > > > > >> >> condition
> > > >> > > > > >> >> > > that would leave an inconsistent state
> > > >> > > > > >> >> > >
> > > >> > > > > >> >> > > commit 162c072e9265a7b6dd9d6f5459eb7974091c4c4e
> > > >> > > > > >> >> > > [FLINK-19492][core] Consolidate Source Events
> > between
> > > >> > Source
> > > >> > > > API
> > > >> > > > > >> and
> > > >> > > > > >> >> > Split
> > > >> > > > > >> >> > > Reader API
> > > >> > > > > >> >> > >
> > > >> > > > > >> >> > > commit c1ca7a4c7c21ec8868c14cdddd43c559625b794c
> > > >> > > > > >> >> > > [refactor][tests] Move some source test utils from
> > > >> > > > > >> >> flink-connector-base
> > > >> > > > > >> >> > to
> > > >> > > > > >> >> > > flink-core
> > > >> > > > > >> >> > >
> > > >> > > > > >> >> > > commit ee5c4c211c35c70d28252363bbc8400453609977
> > > >> > > > > >> >> > > [FLINK-19251][connectors] Avoid confusing queue
> > > >> handling in
> > > >> > > > > >> >> > > "SplitReader.handleSplitsChanges()"
> > > >> > > > > >> >> > >
> > > >> > > > > >> >> > > commit 5abef56b2bf85bcac786f6b16b6899b6cced7176
> > > >> > > > > >> >> > > [FLINK-19250][connectors] Fix error propagation in
> > > >> > connector
> > > >> > > > base
> > > >> > > > > >> >> > > (SplitFetcherManager).
> > > >> > > > > >> >> > >
> > > >> > > > > >> >> > > commit 8fcca837c55a9216595ee4c03038b52747098dbb
> > > >> > > > > >> >> > > [hotfix][connectors] Improve JavaDocs for
> > > >> > > > > >> SingleThreadFetcherManager
> > > >> > > > > >> >> > >
> > > >> > > > > >> >> > > commit 4700bb5dde3303cbe98882f6beb7379425717b01
> > > >> > > > > >> >> > > [FLINK-19225][connectors] Various small
> improvements
> > > to
> > > >> > > > > >> >> SourceReaderBase
> > > >> > > > > >> >> > > (part 2)
> > > >> > > > > >> >> > >
> > > >> > > > > >> >> > > commit 12261c6b7ed6478a9b9f6a69cb58246b83cab9b7
> > > >> > > > > >> >> > > [FLINK-17393][connectors] (follow-up) Wakeup the
> > > >> > > SplitFetchers
> > > >> > > > > more
> > > >> > > > > >> >> > > elegantly.
> > > >> > > > > >> >> > >
> > > >> > > > > >> >> > > commit c60aaff0249bfd6b5871b7f82e03efc487a54d6b
> > > >> > > > > >> >> > > [hotfix][tests] Extend test coverage for
> > > >> > > > > >> FutureCompletingBlockingQueue
> > > >> > > > > >> >> > >
> > > >> > > > > >> >> > > commit cef8a587d7fd2fe64cc644da5ed095d82e46f631
> > > >> > > > > >> >> > > [FLINK-19245][connectors] Set default capacity for
> > > >> > > > > >> >> > > FutureCompletingBlockingQueue.
> > > >> > > > > >> >> > >
> > > >> > > > > >> >> > > commit 4ea95782b4c6a2538153d4d16ad3f4839c7de0fb
> > > >> > > > > >> >> > > [FLINK-19223][connectors] Simplify Availability
> > Future
> > > >> > Model
> > > >> > > in
> > > >> > > > > >> Base
> > > >> > > > > >> >> > > Connector
> > > >> > > > > >> >> > >
> > > >> > > > > >> >> > > commit 511857049ba30c8ff0ee56da551fa4a479dc583e
> > > >> > > > > >> >> > > [FLINK-18128][connectors] Ensure idle split
> fetchers
> > > >> lead
> > > >> > to
> > > >> > > > > >> >> availability
> > > >> > > > > >> >> > > notifications.
> > > >> > > > > >> >> > >
> > > >> > > > > >> >> > > commit a8206467af0830dcb89623ea068b5ca3b3450c92
> > > >> > > > > >> >> > > [refactor][core] Eagerly initialize the FetchTask
> to
> > > >> > support
> > > >> > > > > proper
> > > >> > > > > >> >> unit
> > > >> > > > > >> >> > > testing
> > > >> > > > > >> >> > >
> > > >> > > > > >> >> > > commit 3b2f54bcb437f98e6137c904045cc51072b5c06b
> > > >> > > > > >> >> > > [hotfix][tests] Move constants in SplitFetcherTest
> > > >> relevant
> > > >> > > to
> > > >> > > > > only
> > > >> > > > > >> >> one
> > > >> > > > > >> >> > > test into test method
> > > >> > > > > >> >> > >
> > > >> > > > > >> >> > > commit d7625760a75a508bf05bcddc380bb4d62ee1743e
> > > >> > > > > >> >> > > [FLINK-19225][connectors] Various small
> improvements
> > > to
> > > >> > > > > >> >> SourceReaderBase
> > > >> > > > > >> >> > >
> > > >> > > > > >> >> > > commit a5b0d3297748c1be47ad579a88f24df2255a8df1
> > > >> > > > > >> >> > > [FLINK-17393][connectors] Wakeup the SplitFetchers
> > > more
> > > >> > > > > elegantly.
> > > >> > > > > >> >> > >
> > > >> > > > > >> >> > > commit f42a3ebc3e81a034b7221a803c153636fef34903
> > > >> > > > > >> >> > > [FLINK-18680][connectors] Make connector base
> > > >> > > > RecordsWithSplitIds
> > > >> > > > > >> more
> > > >> > > > > >> >> > > lightweight.
> > > >> > > > > >> >> > >
> > > >> > > > > >> >> > > commit e3d273de822b085183d09b275a445879ff94b350
> > > >> > > > > >> >> > > [FLINK-19162][connectors] Add 'recycle()' to the
> > > >> > > > > >> RecordsWithSplitIds
> > > >> > > > > >> >> to
> > > >> > > > > >> >> > > support reuse of heavy objects.
> > > >> > > > > >> >> > >
> > > >> > > > > >> >> > > commit 8ebc464c2520453a70001cd712abc8dee6ee89e0
> > > >> > > > > >> >> > > [hotfix][testing] Add a set of parameterizable
> > testing
> > > >> > mocks
> > > >> > > > for
> > > >> > > > > >> the
> > > >> > > > > >> >> > Split
> > > >> > > > > >> >> > > Reader API
> > > >> > > > > >> >> > >
> > > >> > > > > >> >> > > commit 930a07438be1185388d7150640f294dfe2a5d378
> > > >> > > > > >> >> > > [hotfix][connectors] Add RequestSplitEvent to
> > > >> > > > > >> 'flink-connector-base'
> > > >> > > > > >> >> > >
> > > >> > > > > >> >> > > commit e945ce8a933bc378844782f784ca473c767ca159
> > > >> > > > > >> >> > > [refactor][connectors] Backport of the
> > connector-base
> > > >> > > exception
> > > >> > > > > >> >> handling
> > > >> > > > > >> >> > > from the Kafka Connector Pull Request
> > > >> > > > > >> >> > >
> > > >> > > > > >> >> > >
> > > >> > > > > >> >> > > On Wed, Oct 28, 2020 at 5:49 PM Steven Wu <
> > > >> > > > stevenz3wu@gmail.com>
> > > >> > > > > >> >> wrote:
> > > >> > > > > >> >> > >
> > > >> > > > > >> >> > > > I would love to see this FLIP-27 source
> interface
> > > >> > > improvement
> > > >> > > > > [1]
> > > >> > > > > >> >> made
> > > >> > > > > >> >> > to
> > > >> > > > > >> >> > > > 1.11.3.
> > > >> > > > > >> >> > > >
> > > >> > > > > >> >> > > > [1]
> > > https://issues.apache.org/jira/browse/FLINK-19698
> > > >> > > > > >> >> > > >
> > > >> > > > > >> >> > > > On Wed, Oct 28, 2020 at 12:32 AM Tzu-Li (Gordon)
> > > Tai <
> > > >> > > > > >> >> > > tzulitai@apache.org>
> > > >> > > > > >> >> > > > wrote:
> > > >> > > > > >> >> > > >
> > > >> > > > > >> >> > > > > Thanks for the replies so far!
> > > >> > > > > >> >> > > > >
> > > >> > > > > >> >> > > > > Just to provide a brief update on the status
> of
> > > >> > blockers
> > > >> > > > for
> > > >> > > > > >> >> 1.11.3
> > > >> > > > > >> >> > so
> > > >> > > > > >> >> > > > far:
> > > >> > > > > >> >> > > > >
> > > >> > > > > >> >> > > > >
> > > >> > > > > >> >> > > > > *PR opened, pending reviewer*- [FLINK-19717]
> > > >> > > > > >> >> > SourceReaderBase.pollNext
> > > >> > > > > >> >> > > > may
> > > >> > > > > >> >> > > > > return END_OF_INPUT if SplitReader.fetch
> throws
> > (
> > > >> > > > > >> >> > > > > https://github.com/apache/flink/pull/13776)
> > > >> > > > > >> >> > > > >
> > > >> > > > > >> >> > > > > *PR opened, reviewed + close to being merged*
> > > >> > > > > >> >> > > > > - [FLINK-19741] Timer service should skip
> > > restoring
> > > >> > from
> > > >> > > > raw
> > > >> > > > > >> keyed
> > > >> > > > > >> >> > > stream
> > > >> > > > > >> >> > > > > if it isn't the writer (
> > > >> > > > > >> >> https://github.com/apache/flink/pull/13761)
> > > >> > > > > >> >> > > > > - [FLINK-19748] Raw keyed stream key group
> > > iterator
> > > >> > > should
> > > >> > > > be
> > > >> > > > > >> >> > skipping
> > > >> > > > > >> >> > > > > unwritten key groups (
> > > >> > > > > >> https://github.com/apache/flink/pull/13772)
> > > >> > > > > >> >> > > > >
> > > >> > > > > >> >> > > > > *Merged*
> > > >> > > > > >> >> > > > > - [FLINK-19154] Application mode deletes HA
> data
> > > in
> > > >> > case
> > > >> > > of
> > > >> > > > > >> >> suspended
> > > >> > > > > >> >> > > > > ZooKeeper connection
> > > >> > > > > >> >> > > > > - [FLINK-19569] Upgrade ICU4J to 67.1+
> > > >> > > > > >> >> > > > >
> > > >> > > > > >> >> > > > > Right now as it seems, progress is mainly
> > blocked
> > > >> on a
> > > >> > > > > reviewer
> > > >> > > > > >> >> for
> > > >> > > > > >> >> > > > > FLINK-19717.
> > > >> > > > > >> >> > > > > Meanwhile, Xintong is keeping an eye on test
> > > >> > > instabilities
> > > >> > > > > [1]
> > > >> > > > > >> to
> > > >> > > > > >> >> see
> > > >> > > > > >> >> > > if
> > > >> > > > > >> >> > > > > there are any fixes that should be applied to
> > > >> > > > `release-1.11`.
> > > >> > > > > >> >> > > > >
> > > >> > > > > >> >> > > > > This is also a reminder, that if there are
> other
> > > >> > blockers
> > > >> > > > > that
> > > >> > > > > >> we
> > > >> > > > > >> >> > need
> > > >> > > > > >> >> > > to
> > > >> > > > > >> >> > > > > be aware of, or a need to re-establish
> estimated
> > > >> time
> > > >> > for
> > > >> > > > > >> getting
> > > >> > > > > >> >> > fixes
> > > >> > > > > >> >> > > > in
> > > >> > > > > >> >> > > > > and delay the RC for 1.11.3, please do let us
> > > know!
> > > >> > > > > >> >> > > > >
> > > >> > > > > >> >> > > > > Cheers,
> > > >> > > > > >> >> > > > > Gordon
> > > >> > > > > >> >> > > > >
> > > >> > > > > >> >> > > > > [1]
> > > >> > > > > >> >> > > > >
> > > >> > > > > >> >> > > > >
> > > >> > > > > >> >> > > >
> > > >> > > > > >> >> > >
> > > >> > > > > >> >> >
> > > >> > > > > >> >>
> > > >> > > > > >>
> > > >> > > > >
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > >
> >
> https://issues.apache.org/jira/issues/?jql=project%20%3D%20FLINK%20AND%20status%20in%20(Open%2C%20%22In%20Progress%22%2C%20Reopened)%20AND%20affectedVersion%20in%20(1.11.0%2C%201.11.1%2C%201.11.2)%20AND%20labels%20%3D%20test-stability
> > > >> > > > > >> >> > > > >
> > > >> > > > > >> >> > > > > On Mon, Oct 26, 2020 at 9:43 PM Kostas
> Kloudas <
> > > >> > > > > >> >> kkloudas@apache.org>
> > > >> > > > > >> >> > > > > wrote:
> > > >> > > > > >> >> > > > >
> > > >> > > > > >> >> > > > > > +1 for releasing Flink 1.11.3 as it
> contains a
> > > >> number
> > > >> > > of
> > > >> > > > > >> >> important
> > > >> > > > > >> >> > > > > > fixes and thanks Gordon and Xintong for
> > > >> volunteering.
> > > >> > > > > >> >> > > > > >
> > > >> > > > > >> >> > > > > > Cheers,
> > > >> > > > > >> >> > > > > > Kostas
> > > >> > > > > >> >> > > > > >
> > > >> > > > > >> >> > > > > > On Mon, Oct 26, 2020 at 4:37 AM Yu Li <
> > > >> > > carp84@gmail.com>
> > > >> > > > > >> wrote:
> > > >> > > > > >> >> > > > > > >
> > > >> > > > > >> >> > > > > > > +1 for releasing Flink 1.11.3, and thanks
> > > Gordon
> > > >> > and
> > > >> > > > > >> Xintong
> > > >> > > > > >> >> for
> > > >> > > > > >> >> > > > > > > volunteering as our release managers.
> > > >> > > > > >> >> > > > > > >
> > > >> > > > > >> >> > > > > > > Best Regards,
> > > >> > > > > >> >> > > > > > > Yu
> > > >> > > > > >> >> > > > > > >
> > > >> > > > > >> >> > > > > > >
> > > >> > > > > >> >> > > > > > > On Mon, 26 Oct 2020 at 09:45, Xintong
> Song <
> > > >> > > > > >> >> > tonysong820@gmail.com>
> > > >> > > > > >> >> > > > > > wrote:
> > > >> > > > > >> >> > > > > > >
> > > >> > > > > >> >> > > > > > > > Thanks Gordan for starting this
> > discussion.
> > > >> > > > > >> >> > > > > > > > My pleasure to help with the release
> > > process.
> > > >> > > > > >> >> > > > > > > >
> > > >> > > > > >> >> > > > > > > > Thank you~
> > > >> > > > > >> >> > > > > > > >
> > > >> > > > > >> >> > > > > > > > Xintong Song
> > > >> > > > > >> >> > > > > > > >
> > > >> > > > > >> >> > > > > > > >
> > > >> > > > > >> >> > > > > > > >
> > > >> > > > > >> >> > > > > > > > On Fri, Oct 23, 2020 at 11:29 PM Till
> > > >> Rohrmann <
> > > >> > > > > >> >> > > > trohrmann@apache.org
> > > >> > > > > >> >> > > > > >
> > > >> > > > > >> >> > > > > > > > wrote:
> > > >> > > > > >> >> > > > > > > >
> > > >> > > > > >> >> > > > > > > > > Thanks for starting this discussion
> > > Gordon.
> > > >> > There
> > > >> > > > are
> > > >> > > > > >> over
> > > >> > > > > >> >> > 100
> > > >> > > > > >> >> > > > > issues
> > > >> > > > > >> >> > > > > > > > > which are fixed for 1.11.3. Hence +1
> > for a
> > > >> > > soonish
> > > >> > > > > >> 1.11.3
> > > >> > > > > >> >> > > > release.
> > > >> > > > > >> >> > > > > > Thanks
> > > >> > > > > >> >> > > > > > > > > for volunteering as our release
> managers
> > > >> Gordon
> > > >> > > and
> > > >> > > > > >> >> Xintong!
> > > >> > > > > >> >> > > > > > > > >
> > > >> > > > > >> >> > > > > > > > > Cheers,
> > > >> > > > > >> >> > > > > > > > > Till
> > > >> > > > > >> >> > > > > > > > >
> > > >> > > > > >> >> > > > > > > > > On Fri, Oct 23, 2020 at 5:02 PM Tzu-Li
> > > >> (Gordon)
> > > >> > > > Tai <
> > > >> > > > > >> >> > > > > > tzulitai@apache.org
> > > >> > > > > >> >> > > > > > > > >
> > > >> > > > > >> >> > > > > > > > > wrote:
> > > >> > > > > >> >> > > > > > > > >
> > > >> > > > > >> >> > > > > > > > >> Hi,
> > > >> > > > > >> >> > > > > > > > >>
> > > >> > > > > >> >> > > > > > > > >> Xintong and I would like to start a
> > > >> discussion
> > > >> > > for
> > > >> > > > > >> >> releasing
> > > >> > > > > >> >> > > > Flink
> > > >> > > > > >> >> > > > > > > > 1.11.3
> > > >> > > > > >> >> > > > > > > > >> soon.
> > > >> > > > > >> >> > > > > > > > >>
> > > >> > > > > >> >> > > > > > > > >> It seems like we already have a few
> > > >> pressing
> > > >> > > > issues
> > > >> > > > > >> that
> > > >> > > > > >> >> > needs
> > > >> > > > > >> >> > > > to
> > > >> > > > > >> >> > > > > be
> > > >> > > > > >> >> > > > > > > > >> included in a new hotfix release:
> > > >> > > > > >> >> > > > > > > > >>
> > > >> > > > > >> >> > > > > > > > >>    - Heap-based timers’ restore
> > behaviour
> > > >> is
> > > >> > > > > causing a
> > > >> > > > > >> >> > > critical
> > > >> > > > > >> >> > > > > > recovery
> > > >> > > > > >> >> > > > > > > > >>    issue for StateFun [1] [2] [3].
> > > >> > > > > >> >> > > > > > > > >>    - There are several robustness
> > issues
> > > >> for
> > > >> > the
> > > >> > > > > >> FLIP-27
> > > >> > > > > >> >> new
> > > >> > > > > >> >> > > > > source
> > > >> > > > > >> >> > > > > > API,
> > > >> > > > > >> >> > > > > > > > >>    such as [4]. We already have some
> > > users
> > > >> > using
> > > >> > > > the
> > > >> > > > > >> >> FLIP-27
> > > >> > > > > >> >> > > API
> > > >> > > > > >> >> > > > > > with
> > > >> > > > > >> >> > > > > > > > >> 1.11.x,
> > > >> > > > > >> >> > > > > > > > >>    so it would be important to get
> > those
> > > >> fixes
> > > >> > > in
> > > >> > > > > for
> > > >> > > > > >> >> 1.11.x
> > > >> > > > > >> >> > > as
> > > >> > > > > >> >> > > > > > well.
> > > >> > > > > >> >> > > > > > > > >>
> > > >> > > > > >> >> > > > > > > > >> Apart from the issues that are
> already
> > > >> marked
> > > >> > as
> > > >> > > > > >> blocker
> > > >> > > > > >> >> for
> > > >> > > > > >> >> > > > > 1.11.3
> > > >> > > > > >> >> > > > > > in
> > > >> > > > > >> >> > > > > > > > our
> > > >> > > > > >> >> > > > > > > > >> JIRA [5], please let us know in this
> > > >> thread if
> > > >> > > > there
> > > >> > > > > >> is
> > > >> > > > > >> >> > > already
> > > >> > > > > >> >> > > > > > ongoing
> > > >> > > > > >> >> > > > > > > > >> work for other important fixes that
> we
> > > >> should
> > > >> > > try
> > > >> > > > to
> > > >> > > > > >> >> > include.
> > > >> > > > > >> >> > > > > > > > >>
> > > >> > > > > >> >> > > > > > > > >> Xintong and I would like to volunteer
> > for
> > > >> > > managing
> > > >> > > > > >> this
> > > >> > > > > >> >> > > release,
> > > >> > > > > >> >> > > > > and
> > > >> > > > > >> >> > > > > > > > will
> > > >> > > > > >> >> > > > > > > > >> try to communicate the priority of
> > > pending
> > > >> > > > blockers
> > > >> > > > > >> over
> > > >> > > > > >> >> the
> > > >> > > > > >> >> > > > next
> > > >> > > > > >> >> > > > > > few
> > > >> > > > > >> >> > > > > > > > >> days.
> > > >> > > > > >> >> > > > > > > > >> Since the aforementioned issues are
> > quite
> > > >> > > > critical,
> > > >> > > > > >> we’d
> > > >> > > > > >> >> > like
> > > >> > > > > >> >> > > to
> > > >> > > > > >> >> > > > > aim
> > > >> > > > > >> >> > > > > > > > >> for a *feature
> > > >> > > > > >> >> > > > > > > > >> freeze by the end of next week (Oct.
> > > 30th)*
> > > >> > and
> > > >> > > > > start
> > > >> > > > > >> the
> > > >> > > > > >> >> > > > release
> > > >> > > > > >> >> > > > > > voting
> > > >> > > > > >> >> > > > > > > > >> process the week after.
> > > >> > > > > >> >> > > > > > > > >> If that is too short of a notice and
> > you
> > > >> might
> > > >> > > > need
> > > >> > > > > >> more
> > > >> > > > > >> >> > time,
> > > >> > > > > >> >> > > > > > please
> > > >> > > > > >> >> > > > > > > > let
> > > >> > > > > >> >> > > > > > > > >> us know!
> > > >> > > > > >> >> > > > > > > > >>
> > > >> > > > > >> >> > > > > > > > >> Cheers,
> > > >> > > > > >> >> > > > > > > > >> Gordon
> > > >> > > > > >> >> > > > > > > > >>
> > > >> > > > > >> >> > > > > > > > >> [1]
> > > >> > > > > https://issues.apache.org/jira/browse/FLINK-19692
> > > >> > > > > >> >> > > > > > > > >> [2]
> > > >> > > > > https://issues.apache.org/jira/browse/FLINK-19741
> > > >> > > > > >> >> > > > > > > > >> [3]
> > > >> > > > > https://issues.apache.org/jira/browse/FLINK-19748
> > > >> > > > > >> >> > > > > > > > >> [4]
> > > >> > > > > https://issues.apache.org/jira/browse/FLINK-19717
> > > >> > > > > >> >> > > > > > > > >> [5]
> > > >> > > > > >> >> > > > > > > > >>
> > > >> > > > > >> >> > > > > > > > >>
> > > >> > > > > >> >> > > > > > > >
> > > >> > > > > >> >> > > > > >
> > > >> > > > > >> >> > > > >
> > > >> > > > > >> >> > > >
> > > >> > > > > >> >> > >
> > > >> > > > > >> >> >
> > > >> > > > > >> >>
> > > >> > > > > >>
> > > >> > > > >
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > >
> >
> https://issues.apache.org/jira/issues/?jql=project%20%3D%20FLINK%20AND%20priority%20%3D%20Blocker%20AND%20fixVersion%20%3D%201.11.3
> > > >> > > > > >> >> > > > > > > > >>
> > > >> > > > > >> >> > > > > > > > >
> > > >> > > > > >> >> > > > > > > >
> > > >> > > > > >> >> > > > > >
> > > >> > > > > >> >> > > > >
> > > >> > > > > >> >> > > >
> > > >> > > > > >> >> > >
> > > >> > > > > >> >> >
> > > >> > > > > >> >> >
> > > >> > > > > >> >> > --
> > > >> > > > > >> >> > Best, Jingsong Lee
> > > >> > > > > >> >> >
> > > >> > > > > >> >>
> > > >> > > > > >> >
> > > >> > > > > >>
> > > >> > > > > >
> > > >> > > > >
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > > >
> > >
> >
>

Re: [DISCUSS] Releasing Apache Flink 1.11.3

Posted by Till Rohrmann <tr...@apache.org>.
FLINK-20033 has been merged.

Cheers,
Till

On Mon, Nov 9, 2020 at 9:39 AM Till Rohrmann <tr...@apache.org> wrote:

> I would like to bring one more critical issue to your attention which is
> FLINK-20033 [1]. I believe that this issue is actually causing what has
> been reported in FLINK-19816 [2]. I hope to have it fixed by the end of
> today. Once FLINK-20033 is fixed, I think that we don't have to block the
> release on FLINK-19816.
>
> [1] https://issues.apache.org/jira/browse/FLINK-20033
> [2] https://issues.apache.org/jira/browse/FLINK-19816
>
> Cheers,
> Till
>
> On Mon, Nov 9, 2020 at 4:05 AM Xintong Song <to...@gmail.com> wrote:
>
>> Hi devs,
>>
>> I'd like to provide an update on the progress of preparing release 1.11.3.
>>
>> *Blockers*
>> We currently have 3 remaining blockers. (3 resolved and 1 emerged compared
>> to last week)
>>
>> - [FLINK-19698] Add close() method and onCheckpointComplete() to the
>> Source.
>> The issue has been fixed on the master branch. It's currently blocked on
>> the FLIP-27 backportings to backport it to the 1.11 branch.
>>
>> - [FLINK-19717] SourceReaderBase.pollNext may return END_OF_INPUT if
>> SplitReader.fetch throws
>> A PR has been opened and reviewed. From the discussions on the PR, it
>> looks
>> close to mergeable.
>>
>> - [FLINK-19816] Flink restored from a wrong checkpoint (a very old one and
>> not the last completed one)
>> This is a newly emerged blocker and Matthias is working on it.
>>
>> *Test Instabilities*
>> We currently have 27 test instabilities[1].
>> AFAIK, none of them are as serious as to block the 1.11.3 release.
>>
>> *FLIP-27 Backprotings*
>>
>> I noticed that there's no jira issues opened on the FLIP-27 backporting
>> efforts, which is part of the major efforts planned for the 1.11.3
>> release,
>> making it hard to track the progress.
>>
>>
>> @Stephan and @Becket, could you please share the updates on the
>> backporting
>> efforts? How is the progress and when are the efforts expected to be
>> finished? It would be appreciated and helpful if we can have a jira ticket
>> to track the backporing progress publicly.
>>
>>
>> Thank you~
>>
>> Xintong Song
>>
>>
>> [1]
>>
>> https://issues.apache.org/jira/browse/FLINK-16908?filter=12348580&jql=project%20%3D%20FLINK%20AND%20resolution%20%3D%20Unresolved%20AND%20affectedVersion%20in%20(1.11.0%2C%201.11.1%2C%201.11.2%2C%201.11.3)%20AND%20labels%20%3D%20test-stability%20ORDER%20BY%20updated%20DESC%2C%20created%20DESC
>>
>> On Wed, Nov 4, 2020 at 3:58 PM Tzu-Li (Gordon) Tai <tz...@apache.org>
>> wrote:
>>
>> > Thanks for the efforts so far with pushing for 1.11.3.
>> >
>> > I'd like to provide a mid-week update on how we're looking with all the
>> > pending blockers and backports:
>> >
>> > *Blockers**:*
>> >
>> > Currently all blockers either have an open PR, or have been merged. I'll
>> > highlight below the blockers *that still require reviewing efforts for
>> them
>> > to move forward:*
>> >
>> > - [FLINK-19909] Flink application in attach mode could not terminate
>> when
>> > the only job is canceled. PR:
>> https://github.com/apache/flink/pull/13911
>> > - [FLINK-19717] SourceReaderBase.pollNext may return END_OF_INPUT if
>> > SplitReader.fetch throws. PR:
>> https://github.com/apache/flink/pull/13776
>> >
>> > The above PRs currently have no reviews at all yet. They seem to already
>> > have designated reviewers.
>> >
>> >
>> > *Backports:*
>> > - FLIP-27 robustness improvement backports: Stephan is currently
>> working on
>> > backporting several FLIP-27 changes. There are no PRs yet for the
>> > backported changes.
>> > - DataStreamUtils.collect() refactorings backport: Steven Wu mentioned
>> to
>> > backport this, but AFAIK this isn't assigned to anyone yet.
>> >
>> > *ETAs*:
>> >
>> > I'd like to request ETAs for the remaining backports, to prevent a
>> creep in
>> > the scope of this bugfix release.
>> >
>> > We already have Flink users that would benefit from fixes that have been
>> > merged to release-1.11 already, so technically speaking the backports
>> > should be considered "nice-to-have" (to the best of my knowledge of the
>> > changes) and could potentially be moved to a follow-up 1.11.4.
>> > Most notably, the Stateful Functions project is already waiting on Flink
>> > 1.11.3 to address critical recovery issues (please see the StateFun
>> 2.2.1
>> > release discussion thread [1]).
>> >
>> > @Stephan Ewen <se...@apache.org> @Becket Qin <be...@gmail.com>
>> could
>> > you provide an ETA for the FLIP-27 backports? It would help to get a
>> better
>> > estimate to decide how we proceed here.
>> >
>> > Cheers,
>> > Gordon
>> >
>> > [1]
>> >
>> >
>> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Releasing-StateFun-hotfix-version-2-2-1-td46239.html
>> >
>> > On Wed, Nov 4, 2020 at 3:16 PM Tzu-Li (Gordon) Tai <tzulitai@apache.org
>> >
>> > wrote:
>> >
>> > > > The collect() utils can be picked back, I see no issue with that
>> (it is
>> > > isolated utilities).
>> > >
>> > > Just checking on all the requested backports mentioned in this thread,
>> > and
>> > > figuring out which ones seem to still be unassigned / open.
>> > >
>> > > Is someone working on backporting
>> > >
>> >
>> https://github.com/apache/flink/commit/09a7a66b7313fea64817fe960a8da1265b428efc
>> > > to release-1.11 at the moment?
>> > >
>> > > On Wed, Nov 4, 2020 at 1:39 AM Steven Wu <st...@gmail.com>
>> wrote:
>> > >
>> > >> @Stephan Ewen <se...@apache.org> yeah, we can do that. don't worry
>> > about
>> > >> it. your earlier email had the perfect explanation on why file source
>> > >> shouldn't be backported.
>> > >>
>> > >> On Tue, Nov 3, 2020 at 3:37 AM Stephan Ewen <se...@apache.org>
>> wrote:
>> > >>
>> > >> > @Steven would it be possible to initially copy some of the code
>> into
>> > the
>> > >> > iceberg source and later replace it by a dependency on the Flink
>> file
>> > >> > source?
>> > >> >
>> > >> > On Mon, Nov 2, 2020 at 8:33 PM Steven Wu <st...@gmail.com>
>> > wrote:
>> > >> >
>> > >> > > Stephan, thanks a lot for explaining the file connector. that
>> makes
>> > >> > sense.
>> > >> > >
>> > >> > > I was asking because we were trying to reuse some of the
>> > >> implementations
>> > >> > in
>> > >> > > the file source for Iceberg source. Flink Iceberg source lives in
>> > the
>> > >> > > Iceberg repo, which is not possible to code against the master
>> > branch
>> > >> of
>> > >> > > the Flink code.
>> > >> > >
>> > >> > > On Mon, Nov 2, 2020 at 3:31 AM Stephan Ewen <se...@apache.org>
>> > wrote:
>> > >> > >
>> > >> > > > Hi Steven!
>> > >> > > >
>> > >> > > > So far there are no plans to pick back the file system
>> connector
>> > >> code.
>> > >> > > This
>> > >> > > > is still evolving and not finalized for 1.12, so I don't feel
>> it
>> > is
>> > >> a
>> > >> > > good
>> > >> > > > candidate to be backported.
>> > >> > > > However, with the base connector changes backported, you
>> should be
>> > >> able
>> > >> > > to
>> > >> > > > run the file connector code from master against 1.11.3.
>> > >> > > >
>> > >> > > > The collect() utils can be picked back, I see no issue with
>> that
>> > >> (it is
>> > >> > > > isolated utilities).
>> > >> > > >
>> > >> > > > Best,
>> > >> > > > Stephan
>> > >> > > >
>> > >> > > >
>> > >> > > > On Mon, Nov 2, 2020 at 3:02 AM Steven Wu <stevenz3wu@gmail.com
>> >
>> > >> wrote:
>> > >> > > >
>> > >> > > > > Basically, it would be great to get the latest code in the
>> > >> > > > > flink-connector-files (FLIP-27).
>> > >> > > > >
>> > >> > > > > On Sat, Oct 31, 2020 at 9:57 AM Steven Wu <
>> stevenz3wu@gmail.com
>> > >
>> > >> > > wrote:
>> > >> > > > >
>> > >> > > > > > Stephan, it will be great if we can also backport the
>> > >> > DataStreamUtils
>> > >> > > > > > related commits that help with collecting output from
>> > unbounded
>> > >> > > > streams.
>> > >> > > > > > e.g.
>> > >> > > > > >
>> > >> > > > > >
>> > >> > > > > >
>> > >> > > > >
>> > >> > > >
>> > >> > >
>> > >> >
>> > >>
>> >
>> https://github.com/apache/flink/commit/09a7a66b7313fea64817fe960a8da1265b428efc
>> > >> > > > > >
>> > >> > > > > > I tried to copy and paste the code to unblock myself. but
>> it
>> > >> > quickly
>> > >> > > > got
>> > >> > > > > > into the rabbit hole of more and more code.
>> > >> > > > > >
>> > >> > > > > > On Fri, Oct 30, 2020 at 11:02 AM Stephan Ewen <
>> > sewen@apache.org
>> > >> >
>> > >> > > > wrote:
>> > >> > > > > >
>> > >> > > > > >> I have started with backporting the source API changes.
>> Some
>> > >> minor
>> > >> > > > > >> conflicts to solve, will need a bit more to finish this.
>> > >> > > > > >>
>> > >> > > > > >> On Fri, Oct 30, 2020 at 7:25 AM Tzu-Li (Gordon) Tai <
>> > >> > > > > tzulitai@apache.org>
>> > >> > > > > >> wrote:
>> > >> > > > > >>
>> > >> > > > > >> > @Stephan Ewen <se...@apache.org>
>> > >> > > > > >> > Are there already plans or ongoing efforts for
>> backporting
>> > >> the
>> > >> > > list
>> > >> > > > of
>> > >> > > > > >> > FLIP-27 changes that you posted?
>> > >> > > > > >> >
>> > >> > > > > >> > On Thu, Oct 29, 2020 at 7:08 PM Xintong Song <
>> > >> > > tonysong820@gmail.com
>> > >> > > > >
>> > >> > > > > >> > wrote:
>> > >> > > > > >> >
>> > >> > > > > >> >> Hi folks,
>> > >> > > > > >> >>
>> > >> > > > > >> >> Just to provide some updates concerning the status on
>> the
>> > >> > > > > >> >> test instabilities.
>> > >> > > > > >> >>
>> > >> > > > > >> >> Currently, we have 30 unresolved tickets labeled with
>> > >> `Affects
>> > >> > > > > Version`
>> > >> > > > > >> >> 1.11.x.
>> > >> > > > > >> >>
>> > >> > > > > >> >>
>> > >> > > > > >>
>> > >> > > > >
>> > >> > > >
>> > >> > >
>> > >> >
>> > >>
>> >
>> https://issues.apache.org/jira/browse/FLINK-19775?filter=12348580&jql=project%20%3D%20FLINK%20AND%20resolution%20%3D%20Unresolved%20AND%20affectedVersion%20in%20(1.11.0%2C%201.11.1%2C%201.11.2%2C%201.11.3)%20AND%20labels%20%3D%20test-stability%20ORDER%20BY%20created%20DESC
>> > >> > > > > >> >>
>> > >> > > > > >> >> Among the 30 tickets, 11 of them are:
>> > >> > > > > >> >> - Have occured in the recent 3 months
>> > >> > > > > >> >> - Not confirmed to be pure testability issues
>> > >> > > > > >> >> - Not confirmed to be rare condition cases
>> > >> > > > > >> >>
>> > >> > > > > >> >> It would be nice if someone familiar with these
>> components
>> > >> can
>> > >> > > > take a
>> > >> > > > > >> look
>> > >> > > > > >> >> into these issues.
>> > >> > > > > >> >>
>> > >> > > > > >> >> - https://issues.apache.org/jira/browse/FLINK-17159
>> (ES6)
>> > >> > > > > >> >> - https://issues.apache.org/jira/browse/FLINK-17912
>> > (Kafka)
>> > >> > > > > >> >> - https://issues.apache.org/jira/browse/FLINK-17949
>> > (Kafka)
>> > >> > > > > >> >> ⁃ https://issues.apache.org/jira/browse/FLINK-18444
>> > (Kafka)
>> > >> > > > > >> >> - https://issues.apache.org/jira/browse/FLINK-18634
>> > (Kafka)
>> > >> > > > > >> >> - https://issues.apache.org/jira/browse/FLINK-18648
>> > (Kafka)
>> > >> > > > > >> >> - https://issues.apache.org/jira/browse/FLINK-18807
>> > (Kafka)
>> > >> > > > > >> >> - https://issues.apache.org/jira/browse/FLINK-19369
>> > >> > > > (BlobClientTest)
>> > >> > > > > >> >> - https://issues.apache.org/jira/browse/FLINK-19436
>> > (TPCDS)
>> > >> > > > > >> >> - https://issues.apache.org/jira/browse/FLINK-19690
>> > >> > > > (Format/Parquet)
>> > >> > > > > >> >> - https://issues.apache.org/jira/browse/FLINK-19775
>> > >> > > > > >> >> (SystemProcessingTimeServiceTest)
>> > >> > > > > >> >>
>> > >> > > > > >> >> Thank you~
>> > >> > > > > >> >>
>> > >> > > > > >> >> Xintong Song
>> > >> > > > > >> >>
>> > >> > > > > >> >>
>> > >> > > > > >> >>
>> > >> > > > > >> >> On Thu, Oct 29, 2020 at 10:21 AM Jingsong Li <
>> > >> > > > jingsonglee0@gmail.com
>> > >> > > > > >
>> > >> > > > > >> >> wrote:
>> > >> > > > > >> >>
>> > >> > > > > >> >> > +1 to backport the FLIP-27 adjustments to 1.11.x.
>> > >> > > > > >> >> >
>> > >> > > > > >> >> > If possible, that would be great. Many people are
>> > looking
>> > >> > > forward
>> > >> > > > > to
>> > >> > > > > >> the
>> > >> > > > > >> >> > FLIP-27 interface, but they don't want to take the
>> risk
>> > to
>> > >> > > > upgrade
>> > >> > > > > to
>> > >> > > > > >> >> 1.12
>> > >> > > > > >> >> > (And wait 1.12). After all, 1.11 is a relatively
>> stable
>> > >> > > version.
>> > >> > > > > >> >> >
>> > >> > > > > >> >> > Best,
>> > >> > > > > >> >> > Jingsong
>> > >> > > > > >> >> >
>> > >> > > > > >> >> > On Thu, Oct 29, 2020 at 1:24 AM Stephan Ewen <
>> > >> > sewen@apache.org
>> > >> > > >
>> > >> > > > > >> wrote:
>> > >> > > > > >> >> >
>> > >> > > > > >> >> > > Thanks for starting this.
>> > >> > > > > >> >> > >
>> > >> > > > > >> >> > > +1 form my side to backport the FLIP-27
>> adjustments to
>> > >> > > 1.11.x.
>> > >> > > > > >> >> > >
>> > >> > > > > >> >> > > There were quite a few changes, and I think we
>> need to
>> > >> > > > > cherry-pick
>> > >> > > > > >> >> them
>> > >> > > > > >> >> > all
>> > >> > > > > >> >> > > to not get some inconsistent mix of changes and
>> many
>> > >> merge
>> > >> > > > > >> conflicts.
>> > >> > > > > >> >> > > I made a list below of what we need to add to
>> > >> > "release-1.11".
>> > >> > > > > >> >> > >
>> > >> > > > > >> >> > > *#### Core Source API Changes to backport (in
>> REVERSE
>> > >> > order)*
>> > >> > > > > >> >> > >
>> > >> > > > > >> >> > >   (Use: "git log
>> > >> > > > > >> >> > >
>> > >> > > >
>> flink-core/src/main/java/org/apache/flink/api/connector/source")
>> > >> > > > > >> >> > >
>> > >> > > > > >> >> > > commit 162c072e9265a7b6dd9d6f5459eb7974091c4c4e
>> > >> > > > > >> >> > > [FLINK-19492][core] Consolidate Source Events
>> between
>> > >> > Source
>> > >> > > > API
>> > >> > > > > >> and
>> > >> > > > > >> >> > Split
>> > >> > > > > >> >> > > Reader API
>> > >> > > > > >> >> > >
>> > >> > > > > >> >> > > commit 428fbbad85379f71a9fbf1a1ab3e3dcfa9006436
>> > >> > > > > >> >> > > [FLINK-19205][core] Add access to configuration and
>> > >> > hostname
>> > >> > > in
>> > >> > > > > the
>> > >> > > > > >> >> > > SourceReaderContext
>> > >> > > > > >> >> > >
>> > >> > > > > >> >> > > commit 723e1790744ddc78d19c7c978442af1383f38d33
>> > >> > > > > >> >> > > [hotfix][core] Add to Source Enumerator convenience
>> > >> methods
>> > >> > > to
>> > >> > > > > >> assign
>> > >> > > > > >> >> > > single split
>> > >> > > > > >> >> > >
>> > >> > > > > >> >> > > *#### Connector Base Changes to Backport (in
>> REVERSE
>> > >> > order)*
>> > >> > > > > >> >> > >
>> > >> > > > > >> >> > >   (Use: "git log
>> > flink-connectors/flink-connector-base")
>> > >> > > > > >> >> > >
>> > >> > > > > >> >> > > commit 401f56fe9d6b0271260edf9787cdcbfe4d03874d
>> > >> > > > > >> >> > > [FLINK-19427][FLINK-19489][tests] Fix test
>> conditions
>> > >> for
>> > >> > > > > >> >> > >
>> > 'SplitFetcherTest.testNotifiesWhenGoingIdleConcurrent()'
>> > >> > > > > >> >> > >
>> > >> > > > > >> >> > > commit 68c5c2ff779d82a1ff81ffaf60d8a1b283797db1
>> > >> > > > > >> >> > > [FLINK-19448][connector base] Explicitly check for
>> > >> > > un-expected
>> > >> > > > > >> >> condition
>> > >> > > > > >> >> > > that would leave an inconsistent state
>> > >> > > > > >> >> > >
>> > >> > > > > >> >> > > commit 162c072e9265a7b6dd9d6f5459eb7974091c4c4e
>> > >> > > > > >> >> > > [FLINK-19492][core] Consolidate Source Events
>> between
>> > >> > Source
>> > >> > > > API
>> > >> > > > > >> and
>> > >> > > > > >> >> > Split
>> > >> > > > > >> >> > > Reader API
>> > >> > > > > >> >> > >
>> > >> > > > > >> >> > > commit c1ca7a4c7c21ec8868c14cdddd43c559625b794c
>> > >> > > > > >> >> > > [refactor][tests] Move some source test utils from
>> > >> > > > > >> >> flink-connector-base
>> > >> > > > > >> >> > to
>> > >> > > > > >> >> > > flink-core
>> > >> > > > > >> >> > >
>> > >> > > > > >> >> > > commit ee5c4c211c35c70d28252363bbc8400453609977
>> > >> > > > > >> >> > > [FLINK-19251][connectors] Avoid confusing queue
>> > >> handling in
>> > >> > > > > >> >> > > "SplitReader.handleSplitsChanges()"
>> > >> > > > > >> >> > >
>> > >> > > > > >> >> > > commit 5abef56b2bf85bcac786f6b16b6899b6cced7176
>> > >> > > > > >> >> > > [FLINK-19250][connectors] Fix error propagation in
>> > >> > connector
>> > >> > > > base
>> > >> > > > > >> >> > > (SplitFetcherManager).
>> > >> > > > > >> >> > >
>> > >> > > > > >> >> > > commit 8fcca837c55a9216595ee4c03038b52747098dbb
>> > >> > > > > >> >> > > [hotfix][connectors] Improve JavaDocs for
>> > >> > > > > >> SingleThreadFetcherManager
>> > >> > > > > >> >> > >
>> > >> > > > > >> >> > > commit 4700bb5dde3303cbe98882f6beb7379425717b01
>> > >> > > > > >> >> > > [FLINK-19225][connectors] Various small
>> improvements
>> > to
>> > >> > > > > >> >> SourceReaderBase
>> > >> > > > > >> >> > > (part 2)
>> > >> > > > > >> >> > >
>> > >> > > > > >> >> > > commit 12261c6b7ed6478a9b9f6a69cb58246b83cab9b7
>> > >> > > > > >> >> > > [FLINK-17393][connectors] (follow-up) Wakeup the
>> > >> > > SplitFetchers
>> > >> > > > > more
>> > >> > > > > >> >> > > elegantly.
>> > >> > > > > >> >> > >
>> > >> > > > > >> >> > > commit c60aaff0249bfd6b5871b7f82e03efc487a54d6b
>> > >> > > > > >> >> > > [hotfix][tests] Extend test coverage for
>> > >> > > > > >> FutureCompletingBlockingQueue
>> > >> > > > > >> >> > >
>> > >> > > > > >> >> > > commit cef8a587d7fd2fe64cc644da5ed095d82e46f631
>> > >> > > > > >> >> > > [FLINK-19245][connectors] Set default capacity for
>> > >> > > > > >> >> > > FutureCompletingBlockingQueue.
>> > >> > > > > >> >> > >
>> > >> > > > > >> >> > > commit 4ea95782b4c6a2538153d4d16ad3f4839c7de0fb
>> > >> > > > > >> >> > > [FLINK-19223][connectors] Simplify Availability
>> Future
>> > >> > Model
>> > >> > > in
>> > >> > > > > >> Base
>> > >> > > > > >> >> > > Connector
>> > >> > > > > >> >> > >
>> > >> > > > > >> >> > > commit 511857049ba30c8ff0ee56da551fa4a479dc583e
>> > >> > > > > >> >> > > [FLINK-18128][connectors] Ensure idle split
>> fetchers
>> > >> lead
>> > >> > to
>> > >> > > > > >> >> availability
>> > >> > > > > >> >> > > notifications.
>> > >> > > > > >> >> > >
>> > >> > > > > >> >> > > commit a8206467af0830dcb89623ea068b5ca3b3450c92
>> > >> > > > > >> >> > > [refactor][core] Eagerly initialize the FetchTask
>> to
>> > >> > support
>> > >> > > > > proper
>> > >> > > > > >> >> unit
>> > >> > > > > >> >> > > testing
>> > >> > > > > >> >> > >
>> > >> > > > > >> >> > > commit 3b2f54bcb437f98e6137c904045cc51072b5c06b
>> > >> > > > > >> >> > > [hotfix][tests] Move constants in SplitFetcherTest
>> > >> relevant
>> > >> > > to
>> > >> > > > > only
>> > >> > > > > >> >> one
>> > >> > > > > >> >> > > test into test method
>> > >> > > > > >> >> > >
>> > >> > > > > >> >> > > commit d7625760a75a508bf05bcddc380bb4d62ee1743e
>> > >> > > > > >> >> > > [FLINK-19225][connectors] Various small
>> improvements
>> > to
>> > >> > > > > >> >> SourceReaderBase
>> > >> > > > > >> >> > >
>> > >> > > > > >> >> > > commit a5b0d3297748c1be47ad579a88f24df2255a8df1
>> > >> > > > > >> >> > > [FLINK-17393][connectors] Wakeup the SplitFetchers
>> > more
>> > >> > > > > elegantly.
>> > >> > > > > >> >> > >
>> > >> > > > > >> >> > > commit f42a3ebc3e81a034b7221a803c153636fef34903
>> > >> > > > > >> >> > > [FLINK-18680][connectors] Make connector base
>> > >> > > > RecordsWithSplitIds
>> > >> > > > > >> more
>> > >> > > > > >> >> > > lightweight.
>> > >> > > > > >> >> > >
>> > >> > > > > >> >> > > commit e3d273de822b085183d09b275a445879ff94b350
>> > >> > > > > >> >> > > [FLINK-19162][connectors] Add 'recycle()' to the
>> > >> > > > > >> RecordsWithSplitIds
>> > >> > > > > >> >> to
>> > >> > > > > >> >> > > support reuse of heavy objects.
>> > >> > > > > >> >> > >
>> > >> > > > > >> >> > > commit 8ebc464c2520453a70001cd712abc8dee6ee89e0
>> > >> > > > > >> >> > > [hotfix][testing] Add a set of parameterizable
>> testing
>> > >> > mocks
>> > >> > > > for
>> > >> > > > > >> the
>> > >> > > > > >> >> > Split
>> > >> > > > > >> >> > > Reader API
>> > >> > > > > >> >> > >
>> > >> > > > > >> >> > > commit 930a07438be1185388d7150640f294dfe2a5d378
>> > >> > > > > >> >> > > [hotfix][connectors] Add RequestSplitEvent to
>> > >> > > > > >> 'flink-connector-base'
>> > >> > > > > >> >> > >
>> > >> > > > > >> >> > > commit e945ce8a933bc378844782f784ca473c767ca159
>> > >> > > > > >> >> > > [refactor][connectors] Backport of the
>> connector-base
>> > >> > > exception
>> > >> > > > > >> >> handling
>> > >> > > > > >> >> > > from the Kafka Connector Pull Request
>> > >> > > > > >> >> > >
>> > >> > > > > >> >> > >
>> > >> > > > > >> >> > > On Wed, Oct 28, 2020 at 5:49 PM Steven Wu <
>> > >> > > > stevenz3wu@gmail.com>
>> > >> > > > > >> >> wrote:
>> > >> > > > > >> >> > >
>> > >> > > > > >> >> > > > I would love to see this FLIP-27 source interface
>> > >> > > improvement
>> > >> > > > > [1]
>> > >> > > > > >> >> made
>> > >> > > > > >> >> > to
>> > >> > > > > >> >> > > > 1.11.3.
>> > >> > > > > >> >> > > >
>> > >> > > > > >> >> > > > [1]
>> > https://issues.apache.org/jira/browse/FLINK-19698
>> > >> > > > > >> >> > > >
>> > >> > > > > >> >> > > > On Wed, Oct 28, 2020 at 12:32 AM Tzu-Li (Gordon)
>> > Tai <
>> > >> > > > > >> >> > > tzulitai@apache.org>
>> > >> > > > > >> >> > > > wrote:
>> > >> > > > > >> >> > > >
>> > >> > > > > >> >> > > > > Thanks for the replies so far!
>> > >> > > > > >> >> > > > >
>> > >> > > > > >> >> > > > > Just to provide a brief update on the status of
>> > >> > blockers
>> > >> > > > for
>> > >> > > > > >> >> 1.11.3
>> > >> > > > > >> >> > so
>> > >> > > > > >> >> > > > far:
>> > >> > > > > >> >> > > > >
>> > >> > > > > >> >> > > > >
>> > >> > > > > >> >> > > > > *PR opened, pending reviewer*- [FLINK-19717]
>> > >> > > > > >> >> > SourceReaderBase.pollNext
>> > >> > > > > >> >> > > > may
>> > >> > > > > >> >> > > > > return END_OF_INPUT if SplitReader.fetch
>> throws (
>> > >> > > > > >> >> > > > > https://github.com/apache/flink/pull/13776)
>> > >> > > > > >> >> > > > >
>> > >> > > > > >> >> > > > > *PR opened, reviewed + close to being merged*
>> > >> > > > > >> >> > > > > - [FLINK-19741] Timer service should skip
>> > restoring
>> > >> > from
>> > >> > > > raw
>> > >> > > > > >> keyed
>> > >> > > > > >> >> > > stream
>> > >> > > > > >> >> > > > > if it isn't the writer (
>> > >> > > > > >> >> https://github.com/apache/flink/pull/13761)
>> > >> > > > > >> >> > > > > - [FLINK-19748] Raw keyed stream key group
>> > iterator
>> > >> > > should
>> > >> > > > be
>> > >> > > > > >> >> > skipping
>> > >> > > > > >> >> > > > > unwritten key groups (
>> > >> > > > > >> https://github.com/apache/flink/pull/13772)
>> > >> > > > > >> >> > > > >
>> > >> > > > > >> >> > > > > *Merged*
>> > >> > > > > >> >> > > > > - [FLINK-19154] Application mode deletes HA
>> data
>> > in
>> > >> > case
>> > >> > > of
>> > >> > > > > >> >> suspended
>> > >> > > > > >> >> > > > > ZooKeeper connection
>> > >> > > > > >> >> > > > > - [FLINK-19569] Upgrade ICU4J to 67.1+
>> > >> > > > > >> >> > > > >
>> > >> > > > > >> >> > > > > Right now as it seems, progress is mainly
>> blocked
>> > >> on a
>> > >> > > > > reviewer
>> > >> > > > > >> >> for
>> > >> > > > > >> >> > > > > FLINK-19717.
>> > >> > > > > >> >> > > > > Meanwhile, Xintong is keeping an eye on test
>> > >> > > instabilities
>> > >> > > > > [1]
>> > >> > > > > >> to
>> > >> > > > > >> >> see
>> > >> > > > > >> >> > > if
>> > >> > > > > >> >> > > > > there are any fixes that should be applied to
>> > >> > > > `release-1.11`.
>> > >> > > > > >> >> > > > >
>> > >> > > > > >> >> > > > > This is also a reminder, that if there are
>> other
>> > >> > blockers
>> > >> > > > > that
>> > >> > > > > >> we
>> > >> > > > > >> >> > need
>> > >> > > > > >> >> > > to
>> > >> > > > > >> >> > > > > be aware of, or a need to re-establish
>> estimated
>> > >> time
>> > >> > for
>> > >> > > > > >> getting
>> > >> > > > > >> >> > fixes
>> > >> > > > > >> >> > > > in
>> > >> > > > > >> >> > > > > and delay the RC for 1.11.3, please do let us
>> > know!
>> > >> > > > > >> >> > > > >
>> > >> > > > > >> >> > > > > Cheers,
>> > >> > > > > >> >> > > > > Gordon
>> > >> > > > > >> >> > > > >
>> > >> > > > > >> >> > > > > [1]
>> > >> > > > > >> >> > > > >
>> > >> > > > > >> >> > > > >
>> > >> > > > > >> >> > > >
>> > >> > > > > >> >> > >
>> > >> > > > > >> >> >
>> > >> > > > > >> >>
>> > >> > > > > >>
>> > >> > > > >
>> > >> > > >
>> > >> > >
>> > >> >
>> > >>
>> >
>> https://issues.apache.org/jira/issues/?jql=project%20%3D%20FLINK%20AND%20status%20in%20(Open%2C%20%22In%20Progress%22%2C%20Reopened)%20AND%20affectedVersion%20in%20(1.11.0%2C%201.11.1%2C%201.11.2)%20AND%20labels%20%3D%20test-stability
>> > >> > > > > >> >> > > > >
>> > >> > > > > >> >> > > > > On Mon, Oct 26, 2020 at 9:43 PM Kostas Kloudas
>> <
>> > >> > > > > >> >> kkloudas@apache.org>
>> > >> > > > > >> >> > > > > wrote:
>> > >> > > > > >> >> > > > >
>> > >> > > > > >> >> > > > > > +1 for releasing Flink 1.11.3 as it contains
>> a
>> > >> number
>> > >> > > of
>> > >> > > > > >> >> important
>> > >> > > > > >> >> > > > > > fixes and thanks Gordon and Xintong for
>> > >> volunteering.
>> > >> > > > > >> >> > > > > >
>> > >> > > > > >> >> > > > > > Cheers,
>> > >> > > > > >> >> > > > > > Kostas
>> > >> > > > > >> >> > > > > >
>> > >> > > > > >> >> > > > > > On Mon, Oct 26, 2020 at 4:37 AM Yu Li <
>> > >> > > carp84@gmail.com>
>> > >> > > > > >> wrote:
>> > >> > > > > >> >> > > > > > >
>> > >> > > > > >> >> > > > > > > +1 for releasing Flink 1.11.3, and thanks
>> > Gordon
>> > >> > and
>> > >> > > > > >> Xintong
>> > >> > > > > >> >> for
>> > >> > > > > >> >> > > > > > > volunteering as our release managers.
>> > >> > > > > >> >> > > > > > >
>> > >> > > > > >> >> > > > > > > Best Regards,
>> > >> > > > > >> >> > > > > > > Yu
>> > >> > > > > >> >> > > > > > >
>> > >> > > > > >> >> > > > > > >
>> > >> > > > > >> >> > > > > > > On Mon, 26 Oct 2020 at 09:45, Xintong Song
>> <
>> > >> > > > > >> >> > tonysong820@gmail.com>
>> > >> > > > > >> >> > > > > > wrote:
>> > >> > > > > >> >> > > > > > >
>> > >> > > > > >> >> > > > > > > > Thanks Gordan for starting this
>> discussion.
>> > >> > > > > >> >> > > > > > > > My pleasure to help with the release
>> > process.
>> > >> > > > > >> >> > > > > > > >
>> > >> > > > > >> >> > > > > > > > Thank you~
>> > >> > > > > >> >> > > > > > > >
>> > >> > > > > >> >> > > > > > > > Xintong Song
>> > >> > > > > >> >> > > > > > > >
>> > >> > > > > >> >> > > > > > > >
>> > >> > > > > >> >> > > > > > > >
>> > >> > > > > >> >> > > > > > > > On Fri, Oct 23, 2020 at 11:29 PM Till
>> > >> Rohrmann <
>> > >> > > > > >> >> > > > trohrmann@apache.org
>> > >> > > > > >> >> > > > > >
>> > >> > > > > >> >> > > > > > > > wrote:
>> > >> > > > > >> >> > > > > > > >
>> > >> > > > > >> >> > > > > > > > > Thanks for starting this discussion
>> > Gordon.
>> > >> > There
>> > >> > > > are
>> > >> > > > > >> over
>> > >> > > > > >> >> > 100
>> > >> > > > > >> >> > > > > issues
>> > >> > > > > >> >> > > > > > > > > which are fixed for 1.11.3. Hence +1
>> for a
>> > >> > > soonish
>> > >> > > > > >> 1.11.3
>> > >> > > > > >> >> > > > release.
>> > >> > > > > >> >> > > > > > Thanks
>> > >> > > > > >> >> > > > > > > > > for volunteering as our release
>> managers
>> > >> Gordon
>> > >> > > and
>> > >> > > > > >> >> Xintong!
>> > >> > > > > >> >> > > > > > > > >
>> > >> > > > > >> >> > > > > > > > > Cheers,
>> > >> > > > > >> >> > > > > > > > > Till
>> > >> > > > > >> >> > > > > > > > >
>> > >> > > > > >> >> > > > > > > > > On Fri, Oct 23, 2020 at 5:02 PM Tzu-Li
>> > >> (Gordon)
>> > >> > > > Tai <
>> > >> > > > > >> >> > > > > > tzulitai@apache.org
>> > >> > > > > >> >> > > > > > > > >
>> > >> > > > > >> >> > > > > > > > > wrote:
>> > >> > > > > >> >> > > > > > > > >
>> > >> > > > > >> >> > > > > > > > >> Hi,
>> > >> > > > > >> >> > > > > > > > >>
>> > >> > > > > >> >> > > > > > > > >> Xintong and I would like to start a
>> > >> discussion
>> > >> > > for
>> > >> > > > > >> >> releasing
>> > >> > > > > >> >> > > > Flink
>> > >> > > > > >> >> > > > > > > > 1.11.3
>> > >> > > > > >> >> > > > > > > > >> soon.
>> > >> > > > > >> >> > > > > > > > >>
>> > >> > > > > >> >> > > > > > > > >> It seems like we already have a few
>> > >> pressing
>> > >> > > > issues
>> > >> > > > > >> that
>> > >> > > > > >> >> > needs
>> > >> > > > > >> >> > > > to
>> > >> > > > > >> >> > > > > be
>> > >> > > > > >> >> > > > > > > > >> included in a new hotfix release:
>> > >> > > > > >> >> > > > > > > > >>
>> > >> > > > > >> >> > > > > > > > >>    - Heap-based timers’ restore
>> behaviour
>> > >> is
>> > >> > > > > causing a
>> > >> > > > > >> >> > > critical
>> > >> > > > > >> >> > > > > > recovery
>> > >> > > > > >> >> > > > > > > > >>    issue for StateFun [1] [2] [3].
>> > >> > > > > >> >> > > > > > > > >>    - There are several robustness
>> issues
>> > >> for
>> > >> > the
>> > >> > > > > >> FLIP-27
>> > >> > > > > >> >> new
>> > >> > > > > >> >> > > > > source
>> > >> > > > > >> >> > > > > > API,
>> > >> > > > > >> >> > > > > > > > >>    such as [4]. We already have some
>> > users
>> > >> > using
>> > >> > > > the
>> > >> > > > > >> >> FLIP-27
>> > >> > > > > >> >> > > API
>> > >> > > > > >> >> > > > > > with
>> > >> > > > > >> >> > > > > > > > >> 1.11.x,
>> > >> > > > > >> >> > > > > > > > >>    so it would be important to get
>> those
>> > >> fixes
>> > >> > > in
>> > >> > > > > for
>> > >> > > > > >> >> 1.11.x
>> > >> > > > > >> >> > > as
>> > >> > > > > >> >> > > > > > well.
>> > >> > > > > >> >> > > > > > > > >>
>> > >> > > > > >> >> > > > > > > > >> Apart from the issues that are already
>> > >> marked
>> > >> > as
>> > >> > > > > >> blocker
>> > >> > > > > >> >> for
>> > >> > > > > >> >> > > > > 1.11.3
>> > >> > > > > >> >> > > > > > in
>> > >> > > > > >> >> > > > > > > > our
>> > >> > > > > >> >> > > > > > > > >> JIRA [5], please let us know in this
>> > >> thread if
>> > >> > > > there
>> > >> > > > > >> is
>> > >> > > > > >> >> > > already
>> > >> > > > > >> >> > > > > > ongoing
>> > >> > > > > >> >> > > > > > > > >> work for other important fixes that we
>> > >> should
>> > >> > > try
>> > >> > > > to
>> > >> > > > > >> >> > include.
>> > >> > > > > >> >> > > > > > > > >>
>> > >> > > > > >> >> > > > > > > > >> Xintong and I would like to volunteer
>> for
>> > >> > > managing
>> > >> > > > > >> this
>> > >> > > > > >> >> > > release,
>> > >> > > > > >> >> > > > > and
>> > >> > > > > >> >> > > > > > > > will
>> > >> > > > > >> >> > > > > > > > >> try to communicate the priority of
>> > pending
>> > >> > > > blockers
>> > >> > > > > >> over
>> > >> > > > > >> >> the
>> > >> > > > > >> >> > > > next
>> > >> > > > > >> >> > > > > > few
>> > >> > > > > >> >> > > > > > > > >> days.
>> > >> > > > > >> >> > > > > > > > >> Since the aforementioned issues are
>> quite
>> > >> > > > critical,
>> > >> > > > > >> we’d
>> > >> > > > > >> >> > like
>> > >> > > > > >> >> > > to
>> > >> > > > > >> >> > > > > aim
>> > >> > > > > >> >> > > > > > > > >> for a *feature
>> > >> > > > > >> >> > > > > > > > >> freeze by the end of next week (Oct.
>> > 30th)*
>> > >> > and
>> > >> > > > > start
>> > >> > > > > >> the
>> > >> > > > > >> >> > > > release
>> > >> > > > > >> >> > > > > > voting
>> > >> > > > > >> >> > > > > > > > >> process the week after.
>> > >> > > > > >> >> > > > > > > > >> If that is too short of a notice and
>> you
>> > >> might
>> > >> > > > need
>> > >> > > > > >> more
>> > >> > > > > >> >> > time,
>> > >> > > > > >> >> > > > > > please
>> > >> > > > > >> >> > > > > > > > let
>> > >> > > > > >> >> > > > > > > > >> us know!
>> > >> > > > > >> >> > > > > > > > >>
>> > >> > > > > >> >> > > > > > > > >> Cheers,
>> > >> > > > > >> >> > > > > > > > >> Gordon
>> > >> > > > > >> >> > > > > > > > >>
>> > >> > > > > >> >> > > > > > > > >> [1]
>> > >> > > > > https://issues.apache.org/jira/browse/FLINK-19692
>> > >> > > > > >> >> > > > > > > > >> [2]
>> > >> > > > > https://issues.apache.org/jira/browse/FLINK-19741
>> > >> > > > > >> >> > > > > > > > >> [3]
>> > >> > > > > https://issues.apache.org/jira/browse/FLINK-19748
>> > >> > > > > >> >> > > > > > > > >> [4]
>> > >> > > > > https://issues.apache.org/jira/browse/FLINK-19717
>> > >> > > > > >> >> > > > > > > > >> [5]
>> > >> > > > > >> >> > > > > > > > >>
>> > >> > > > > >> >> > > > > > > > >>
>> > >> > > > > >> >> > > > > > > >
>> > >> > > > > >> >> > > > > >
>> > >> > > > > >> >> > > > >
>> > >> > > > > >> >> > > >
>> > >> > > > > >> >> > >
>> > >> > > > > >> >> >
>> > >> > > > > >> >>
>> > >> > > > > >>
>> > >> > > > >
>> > >> > > >
>> > >> > >
>> > >> >
>> > >>
>> >
>> https://issues.apache.org/jira/issues/?jql=project%20%3D%20FLINK%20AND%20priority%20%3D%20Blocker%20AND%20fixVersion%20%3D%201.11.3
>> > >> > > > > >> >> > > > > > > > >>
>> > >> > > > > >> >> > > > > > > > >
>> > >> > > > > >> >> > > > > > > >
>> > >> > > > > >> >> > > > > >
>> > >> > > > > >> >> > > > >
>> > >> > > > > >> >> > > >
>> > >> > > > > >> >> > >
>> > >> > > > > >> >> >
>> > >> > > > > >> >> >
>> > >> > > > > >> >> > --
>> > >> > > > > >> >> > Best, Jingsong Lee
>> > >> > > > > >> >> >
>> > >> > > > > >> >>
>> > >> > > > > >> >
>> > >> > > > > >>
>> > >> > > > > >
>> > >> > > > >
>> > >> > > >
>> > >> > >
>> > >> >
>> > >>
>> > >
>> >
>>
>

Re: [DISCUSS] Releasing Apache Flink 1.11.3

Posted by Till Rohrmann <tr...@apache.org>.
I would like to bring one more critical issue to your attention which is
FLINK-20033 [1]. I believe that this issue is actually causing what has
been reported in FLINK-19816 [2]. I hope to have it fixed by the end of
today. Once FLINK-20033 is fixed, I think that we don't have to block the
release on FLINK-19816.

[1] https://issues.apache.org/jira/browse/FLINK-20033
[2] https://issues.apache.org/jira/browse/FLINK-19816

Cheers,
Till

On Mon, Nov 9, 2020 at 4:05 AM Xintong Song <to...@gmail.com> wrote:

> Hi devs,
>
> I'd like to provide an update on the progress of preparing release 1.11.3.
>
> *Blockers*
> We currently have 3 remaining blockers. (3 resolved and 1 emerged compared
> to last week)
>
> - [FLINK-19698] Add close() method and onCheckpointComplete() to the
> Source.
> The issue has been fixed on the master branch. It's currently blocked on
> the FLIP-27 backportings to backport it to the 1.11 branch.
>
> - [FLINK-19717] SourceReaderBase.pollNext may return END_OF_INPUT if
> SplitReader.fetch throws
> A PR has been opened and reviewed. From the discussions on the PR, it looks
> close to mergeable.
>
> - [FLINK-19816] Flink restored from a wrong checkpoint (a very old one and
> not the last completed one)
> This is a newly emerged blocker and Matthias is working on it.
>
> *Test Instabilities*
> We currently have 27 test instabilities[1].
> AFAIK, none of them are as serious as to block the 1.11.3 release.
>
> *FLIP-27 Backprotings*
>
> I noticed that there's no jira issues opened on the FLIP-27 backporting
> efforts, which is part of the major efforts planned for the 1.11.3 release,
> making it hard to track the progress.
>
>
> @Stephan and @Becket, could you please share the updates on the backporting
> efforts? How is the progress and when are the efforts expected to be
> finished? It would be appreciated and helpful if we can have a jira ticket
> to track the backporing progress publicly.
>
>
> Thank you~
>
> Xintong Song
>
>
> [1]
>
> https://issues.apache.org/jira/browse/FLINK-16908?filter=12348580&jql=project%20%3D%20FLINK%20AND%20resolution%20%3D%20Unresolved%20AND%20affectedVersion%20in%20(1.11.0%2C%201.11.1%2C%201.11.2%2C%201.11.3)%20AND%20labels%20%3D%20test-stability%20ORDER%20BY%20updated%20DESC%2C%20created%20DESC
>
> On Wed, Nov 4, 2020 at 3:58 PM Tzu-Li (Gordon) Tai <tz...@apache.org>
> wrote:
>
> > Thanks for the efforts so far with pushing for 1.11.3.
> >
> > I'd like to provide a mid-week update on how we're looking with all the
> > pending blockers and backports:
> >
> > *Blockers**:*
> >
> > Currently all blockers either have an open PR, or have been merged. I'll
> > highlight below the blockers *that still require reviewing efforts for
> them
> > to move forward:*
> >
> > - [FLINK-19909] Flink application in attach mode could not terminate when
> > the only job is canceled. PR: https://github.com/apache/flink/pull/13911
> > - [FLINK-19717] SourceReaderBase.pollNext may return END_OF_INPUT if
> > SplitReader.fetch throws. PR: https://github.com/apache/flink/pull/13776
> >
> > The above PRs currently have no reviews at all yet. They seem to already
> > have designated reviewers.
> >
> >
> > *Backports:*
> > - FLIP-27 robustness improvement backports: Stephan is currently working
> on
> > backporting several FLIP-27 changes. There are no PRs yet for the
> > backported changes.
> > - DataStreamUtils.collect() refactorings backport: Steven Wu mentioned to
> > backport this, but AFAIK this isn't assigned to anyone yet.
> >
> > *ETAs*:
> >
> > I'd like to request ETAs for the remaining backports, to prevent a creep
> in
> > the scope of this bugfix release.
> >
> > We already have Flink users that would benefit from fixes that have been
> > merged to release-1.11 already, so technically speaking the backports
> > should be considered "nice-to-have" (to the best of my knowledge of the
> > changes) and could potentially be moved to a follow-up 1.11.4.
> > Most notably, the Stateful Functions project is already waiting on Flink
> > 1.11.3 to address critical recovery issues (please see the StateFun 2.2.1
> > release discussion thread [1]).
> >
> > @Stephan Ewen <se...@apache.org> @Becket Qin <be...@gmail.com>
> could
> > you provide an ETA for the FLIP-27 backports? It would help to get a
> better
> > estimate to decide how we proceed here.
> >
> > Cheers,
> > Gordon
> >
> > [1]
> >
> >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Releasing-StateFun-hotfix-version-2-2-1-td46239.html
> >
> > On Wed, Nov 4, 2020 at 3:16 PM Tzu-Li (Gordon) Tai <tz...@apache.org>
> > wrote:
> >
> > > > The collect() utils can be picked back, I see no issue with that (it
> is
> > > isolated utilities).
> > >
> > > Just checking on all the requested backports mentioned in this thread,
> > and
> > > figuring out which ones seem to still be unassigned / open.
> > >
> > > Is someone working on backporting
> > >
> >
> https://github.com/apache/flink/commit/09a7a66b7313fea64817fe960a8da1265b428efc
> > > to release-1.11 at the moment?
> > >
> > > On Wed, Nov 4, 2020 at 1:39 AM Steven Wu <st...@gmail.com> wrote:
> > >
> > >> @Stephan Ewen <se...@apache.org> yeah, we can do that. don't worry
> > about
> > >> it. your earlier email had the perfect explanation on why file source
> > >> shouldn't be backported.
> > >>
> > >> On Tue, Nov 3, 2020 at 3:37 AM Stephan Ewen <se...@apache.org> wrote:
> > >>
> > >> > @Steven would it be possible to initially copy some of the code into
> > the
> > >> > iceberg source and later replace it by a dependency on the Flink
> file
> > >> > source?
> > >> >
> > >> > On Mon, Nov 2, 2020 at 8:33 PM Steven Wu <st...@gmail.com>
> > wrote:
> > >> >
> > >> > > Stephan, thanks a lot for explaining the file connector. that
> makes
> > >> > sense.
> > >> > >
> > >> > > I was asking because we were trying to reuse some of the
> > >> implementations
> > >> > in
> > >> > > the file source for Iceberg source. Flink Iceberg source lives in
> > the
> > >> > > Iceberg repo, which is not possible to code against the master
> > branch
> > >> of
> > >> > > the Flink code.
> > >> > >
> > >> > > On Mon, Nov 2, 2020 at 3:31 AM Stephan Ewen <se...@apache.org>
> > wrote:
> > >> > >
> > >> > > > Hi Steven!
> > >> > > >
> > >> > > > So far there are no plans to pick back the file system connector
> > >> code.
> > >> > > This
> > >> > > > is still evolving and not finalized for 1.12, so I don't feel it
> > is
> > >> a
> > >> > > good
> > >> > > > candidate to be backported.
> > >> > > > However, with the base connector changes backported, you should
> be
> > >> able
> > >> > > to
> > >> > > > run the file connector code from master against 1.11.3.
> > >> > > >
> > >> > > > The collect() utils can be picked back, I see no issue with that
> > >> (it is
> > >> > > > isolated utilities).
> > >> > > >
> > >> > > > Best,
> > >> > > > Stephan
> > >> > > >
> > >> > > >
> > >> > > > On Mon, Nov 2, 2020 at 3:02 AM Steven Wu <st...@gmail.com>
> > >> wrote:
> > >> > > >
> > >> > > > > Basically, it would be great to get the latest code in the
> > >> > > > > flink-connector-files (FLIP-27).
> > >> > > > >
> > >> > > > > On Sat, Oct 31, 2020 at 9:57 AM Steven Wu <
> stevenz3wu@gmail.com
> > >
> > >> > > wrote:
> > >> > > > >
> > >> > > > > > Stephan, it will be great if we can also backport the
> > >> > DataStreamUtils
> > >> > > > > > related commits that help with collecting output from
> > unbounded
> > >> > > > streams.
> > >> > > > > > e.g.
> > >> > > > > >
> > >> > > > > >
> > >> > > > > >
> > >> > > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> >
> https://github.com/apache/flink/commit/09a7a66b7313fea64817fe960a8da1265b428efc
> > >> > > > > >
> > >> > > > > > I tried to copy and paste the code to unblock myself. but it
> > >> > quickly
> > >> > > > got
> > >> > > > > > into the rabbit hole of more and more code.
> > >> > > > > >
> > >> > > > > > On Fri, Oct 30, 2020 at 11:02 AM Stephan Ewen <
> > sewen@apache.org
> > >> >
> > >> > > > wrote:
> > >> > > > > >
> > >> > > > > >> I have started with backporting the source API changes.
> Some
> > >> minor
> > >> > > > > >> conflicts to solve, will need a bit more to finish this.
> > >> > > > > >>
> > >> > > > > >> On Fri, Oct 30, 2020 at 7:25 AM Tzu-Li (Gordon) Tai <
> > >> > > > > tzulitai@apache.org>
> > >> > > > > >> wrote:
> > >> > > > > >>
> > >> > > > > >> > @Stephan Ewen <se...@apache.org>
> > >> > > > > >> > Are there already plans or ongoing efforts for
> backporting
> > >> the
> > >> > > list
> > >> > > > of
> > >> > > > > >> > FLIP-27 changes that you posted?
> > >> > > > > >> >
> > >> > > > > >> > On Thu, Oct 29, 2020 at 7:08 PM Xintong Song <
> > >> > > tonysong820@gmail.com
> > >> > > > >
> > >> > > > > >> > wrote:
> > >> > > > > >> >
> > >> > > > > >> >> Hi folks,
> > >> > > > > >> >>
> > >> > > > > >> >> Just to provide some updates concerning the status on
> the
> > >> > > > > >> >> test instabilities.
> > >> > > > > >> >>
> > >> > > > > >> >> Currently, we have 30 unresolved tickets labeled with
> > >> `Affects
> > >> > > > > Version`
> > >> > > > > >> >> 1.11.x.
> > >> > > > > >> >>
> > >> > > > > >> >>
> > >> > > > > >>
> > >> > > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> >
> https://issues.apache.org/jira/browse/FLINK-19775?filter=12348580&jql=project%20%3D%20FLINK%20AND%20resolution%20%3D%20Unresolved%20AND%20affectedVersion%20in%20(1.11.0%2C%201.11.1%2C%201.11.2%2C%201.11.3)%20AND%20labels%20%3D%20test-stability%20ORDER%20BY%20created%20DESC
> > >> > > > > >> >>
> > >> > > > > >> >> Among the 30 tickets, 11 of them are:
> > >> > > > > >> >> - Have occured in the recent 3 months
> > >> > > > > >> >> - Not confirmed to be pure testability issues
> > >> > > > > >> >> - Not confirmed to be rare condition cases
> > >> > > > > >> >>
> > >> > > > > >> >> It would be nice if someone familiar with these
> components
> > >> can
> > >> > > > take a
> > >> > > > > >> look
> > >> > > > > >> >> into these issues.
> > >> > > > > >> >>
> > >> > > > > >> >> - https://issues.apache.org/jira/browse/FLINK-17159
> (ES6)
> > >> > > > > >> >> - https://issues.apache.org/jira/browse/FLINK-17912
> > (Kafka)
> > >> > > > > >> >> - https://issues.apache.org/jira/browse/FLINK-17949
> > (Kafka)
> > >> > > > > >> >> ⁃ https://issues.apache.org/jira/browse/FLINK-18444
> > (Kafka)
> > >> > > > > >> >> - https://issues.apache.org/jira/browse/FLINK-18634
> > (Kafka)
> > >> > > > > >> >> - https://issues.apache.org/jira/browse/FLINK-18648
> > (Kafka)
> > >> > > > > >> >> - https://issues.apache.org/jira/browse/FLINK-18807
> > (Kafka)
> > >> > > > > >> >> - https://issues.apache.org/jira/browse/FLINK-19369
> > >> > > > (BlobClientTest)
> > >> > > > > >> >> - https://issues.apache.org/jira/browse/FLINK-19436
> > (TPCDS)
> > >> > > > > >> >> - https://issues.apache.org/jira/browse/FLINK-19690
> > >> > > > (Format/Parquet)
> > >> > > > > >> >> - https://issues.apache.org/jira/browse/FLINK-19775
> > >> > > > > >> >> (SystemProcessingTimeServiceTest)
> > >> > > > > >> >>
> > >> > > > > >> >> Thank you~
> > >> > > > > >> >>
> > >> > > > > >> >> Xintong Song
> > >> > > > > >> >>
> > >> > > > > >> >>
> > >> > > > > >> >>
> > >> > > > > >> >> On Thu, Oct 29, 2020 at 10:21 AM Jingsong Li <
> > >> > > > jingsonglee0@gmail.com
> > >> > > > > >
> > >> > > > > >> >> wrote:
> > >> > > > > >> >>
> > >> > > > > >> >> > +1 to backport the FLIP-27 adjustments to 1.11.x.
> > >> > > > > >> >> >
> > >> > > > > >> >> > If possible, that would be great. Many people are
> > looking
> > >> > > forward
> > >> > > > > to
> > >> > > > > >> the
> > >> > > > > >> >> > FLIP-27 interface, but they don't want to take the
> risk
> > to
> > >> > > > upgrade
> > >> > > > > to
> > >> > > > > >> >> 1.12
> > >> > > > > >> >> > (And wait 1.12). After all, 1.11 is a relatively
> stable
> > >> > > version.
> > >> > > > > >> >> >
> > >> > > > > >> >> > Best,
> > >> > > > > >> >> > Jingsong
> > >> > > > > >> >> >
> > >> > > > > >> >> > On Thu, Oct 29, 2020 at 1:24 AM Stephan Ewen <
> > >> > sewen@apache.org
> > >> > > >
> > >> > > > > >> wrote:
> > >> > > > > >> >> >
> > >> > > > > >> >> > > Thanks for starting this.
> > >> > > > > >> >> > >
> > >> > > > > >> >> > > +1 form my side to backport the FLIP-27 adjustments
> to
> > >> > > 1.11.x.
> > >> > > > > >> >> > >
> > >> > > > > >> >> > > There were quite a few changes, and I think we need
> to
> > >> > > > > cherry-pick
> > >> > > > > >> >> them
> > >> > > > > >> >> > all
> > >> > > > > >> >> > > to not get some inconsistent mix of changes and many
> > >> merge
> > >> > > > > >> conflicts.
> > >> > > > > >> >> > > I made a list below of what we need to add to
> > >> > "release-1.11".
> > >> > > > > >> >> > >
> > >> > > > > >> >> > > *#### Core Source API Changes to backport (in
> REVERSE
> > >> > order)*
> > >> > > > > >> >> > >
> > >> > > > > >> >> > >   (Use: "git log
> > >> > > > > >> >> > >
> > >> > > > flink-core/src/main/java/org/apache/flink/api/connector/source")
> > >> > > > > >> >> > >
> > >> > > > > >> >> > > commit 162c072e9265a7b6dd9d6f5459eb7974091c4c4e
> > >> > > > > >> >> > > [FLINK-19492][core] Consolidate Source Events
> between
> > >> > Source
> > >> > > > API
> > >> > > > > >> and
> > >> > > > > >> >> > Split
> > >> > > > > >> >> > > Reader API
> > >> > > > > >> >> > >
> > >> > > > > >> >> > > commit 428fbbad85379f71a9fbf1a1ab3e3dcfa9006436
> > >> > > > > >> >> > > [FLINK-19205][core] Add access to configuration and
> > >> > hostname
> > >> > > in
> > >> > > > > the
> > >> > > > > >> >> > > SourceReaderContext
> > >> > > > > >> >> > >
> > >> > > > > >> >> > > commit 723e1790744ddc78d19c7c978442af1383f38d33
> > >> > > > > >> >> > > [hotfix][core] Add to Source Enumerator convenience
> > >> methods
> > >> > > to
> > >> > > > > >> assign
> > >> > > > > >> >> > > single split
> > >> > > > > >> >> > >
> > >> > > > > >> >> > > *#### Connector Base Changes to Backport (in REVERSE
> > >> > order)*
> > >> > > > > >> >> > >
> > >> > > > > >> >> > >   (Use: "git log
> > flink-connectors/flink-connector-base")
> > >> > > > > >> >> > >
> > >> > > > > >> >> > > commit 401f56fe9d6b0271260edf9787cdcbfe4d03874d
> > >> > > > > >> >> > > [FLINK-19427][FLINK-19489][tests] Fix test
> conditions
> > >> for
> > >> > > > > >> >> > >
> > 'SplitFetcherTest.testNotifiesWhenGoingIdleConcurrent()'
> > >> > > > > >> >> > >
> > >> > > > > >> >> > > commit 68c5c2ff779d82a1ff81ffaf60d8a1b283797db1
> > >> > > > > >> >> > > [FLINK-19448][connector base] Explicitly check for
> > >> > > un-expected
> > >> > > > > >> >> condition
> > >> > > > > >> >> > > that would leave an inconsistent state
> > >> > > > > >> >> > >
> > >> > > > > >> >> > > commit 162c072e9265a7b6dd9d6f5459eb7974091c4c4e
> > >> > > > > >> >> > > [FLINK-19492][core] Consolidate Source Events
> between
> > >> > Source
> > >> > > > API
> > >> > > > > >> and
> > >> > > > > >> >> > Split
> > >> > > > > >> >> > > Reader API
> > >> > > > > >> >> > >
> > >> > > > > >> >> > > commit c1ca7a4c7c21ec8868c14cdddd43c559625b794c
> > >> > > > > >> >> > > [refactor][tests] Move some source test utils from
> > >> > > > > >> >> flink-connector-base
> > >> > > > > >> >> > to
> > >> > > > > >> >> > > flink-core
> > >> > > > > >> >> > >
> > >> > > > > >> >> > > commit ee5c4c211c35c70d28252363bbc8400453609977
> > >> > > > > >> >> > > [FLINK-19251][connectors] Avoid confusing queue
> > >> handling in
> > >> > > > > >> >> > > "SplitReader.handleSplitsChanges()"
> > >> > > > > >> >> > >
> > >> > > > > >> >> > > commit 5abef56b2bf85bcac786f6b16b6899b6cced7176
> > >> > > > > >> >> > > [FLINK-19250][connectors] Fix error propagation in
> > >> > connector
> > >> > > > base
> > >> > > > > >> >> > > (SplitFetcherManager).
> > >> > > > > >> >> > >
> > >> > > > > >> >> > > commit 8fcca837c55a9216595ee4c03038b52747098dbb
> > >> > > > > >> >> > > [hotfix][connectors] Improve JavaDocs for
> > >> > > > > >> SingleThreadFetcherManager
> > >> > > > > >> >> > >
> > >> > > > > >> >> > > commit 4700bb5dde3303cbe98882f6beb7379425717b01
> > >> > > > > >> >> > > [FLINK-19225][connectors] Various small improvements
> > to
> > >> > > > > >> >> SourceReaderBase
> > >> > > > > >> >> > > (part 2)
> > >> > > > > >> >> > >
> > >> > > > > >> >> > > commit 12261c6b7ed6478a9b9f6a69cb58246b83cab9b7
> > >> > > > > >> >> > > [FLINK-17393][connectors] (follow-up) Wakeup the
> > >> > > SplitFetchers
> > >> > > > > more
> > >> > > > > >> >> > > elegantly.
> > >> > > > > >> >> > >
> > >> > > > > >> >> > > commit c60aaff0249bfd6b5871b7f82e03efc487a54d6b
> > >> > > > > >> >> > > [hotfix][tests] Extend test coverage for
> > >> > > > > >> FutureCompletingBlockingQueue
> > >> > > > > >> >> > >
> > >> > > > > >> >> > > commit cef8a587d7fd2fe64cc644da5ed095d82e46f631
> > >> > > > > >> >> > > [FLINK-19245][connectors] Set default capacity for
> > >> > > > > >> >> > > FutureCompletingBlockingQueue.
> > >> > > > > >> >> > >
> > >> > > > > >> >> > > commit 4ea95782b4c6a2538153d4d16ad3f4839c7de0fb
> > >> > > > > >> >> > > [FLINK-19223][connectors] Simplify Availability
> Future
> > >> > Model
> > >> > > in
> > >> > > > > >> Base
> > >> > > > > >> >> > > Connector
> > >> > > > > >> >> > >
> > >> > > > > >> >> > > commit 511857049ba30c8ff0ee56da551fa4a479dc583e
> > >> > > > > >> >> > > [FLINK-18128][connectors] Ensure idle split fetchers
> > >> lead
> > >> > to
> > >> > > > > >> >> availability
> > >> > > > > >> >> > > notifications.
> > >> > > > > >> >> > >
> > >> > > > > >> >> > > commit a8206467af0830dcb89623ea068b5ca3b3450c92
> > >> > > > > >> >> > > [refactor][core] Eagerly initialize the FetchTask to
> > >> > support
> > >> > > > > proper
> > >> > > > > >> >> unit
> > >> > > > > >> >> > > testing
> > >> > > > > >> >> > >
> > >> > > > > >> >> > > commit 3b2f54bcb437f98e6137c904045cc51072b5c06b
> > >> > > > > >> >> > > [hotfix][tests] Move constants in SplitFetcherTest
> > >> relevant
> > >> > > to
> > >> > > > > only
> > >> > > > > >> >> one
> > >> > > > > >> >> > > test into test method
> > >> > > > > >> >> > >
> > >> > > > > >> >> > > commit d7625760a75a508bf05bcddc380bb4d62ee1743e
> > >> > > > > >> >> > > [FLINK-19225][connectors] Various small improvements
> > to
> > >> > > > > >> >> SourceReaderBase
> > >> > > > > >> >> > >
> > >> > > > > >> >> > > commit a5b0d3297748c1be47ad579a88f24df2255a8df1
> > >> > > > > >> >> > > [FLINK-17393][connectors] Wakeup the SplitFetchers
> > more
> > >> > > > > elegantly.
> > >> > > > > >> >> > >
> > >> > > > > >> >> > > commit f42a3ebc3e81a034b7221a803c153636fef34903
> > >> > > > > >> >> > > [FLINK-18680][connectors] Make connector base
> > >> > > > RecordsWithSplitIds
> > >> > > > > >> more
> > >> > > > > >> >> > > lightweight.
> > >> > > > > >> >> > >
> > >> > > > > >> >> > > commit e3d273de822b085183d09b275a445879ff94b350
> > >> > > > > >> >> > > [FLINK-19162][connectors] Add 'recycle()' to the
> > >> > > > > >> RecordsWithSplitIds
> > >> > > > > >> >> to
> > >> > > > > >> >> > > support reuse of heavy objects.
> > >> > > > > >> >> > >
> > >> > > > > >> >> > > commit 8ebc464c2520453a70001cd712abc8dee6ee89e0
> > >> > > > > >> >> > > [hotfix][testing] Add a set of parameterizable
> testing
> > >> > mocks
> > >> > > > for
> > >> > > > > >> the
> > >> > > > > >> >> > Split
> > >> > > > > >> >> > > Reader API
> > >> > > > > >> >> > >
> > >> > > > > >> >> > > commit 930a07438be1185388d7150640f294dfe2a5d378
> > >> > > > > >> >> > > [hotfix][connectors] Add RequestSplitEvent to
> > >> > > > > >> 'flink-connector-base'
> > >> > > > > >> >> > >
> > >> > > > > >> >> > > commit e945ce8a933bc378844782f784ca473c767ca159
> > >> > > > > >> >> > > [refactor][connectors] Backport of the
> connector-base
> > >> > > exception
> > >> > > > > >> >> handling
> > >> > > > > >> >> > > from the Kafka Connector Pull Request
> > >> > > > > >> >> > >
> > >> > > > > >> >> > >
> > >> > > > > >> >> > > On Wed, Oct 28, 2020 at 5:49 PM Steven Wu <
> > >> > > > stevenz3wu@gmail.com>
> > >> > > > > >> >> wrote:
> > >> > > > > >> >> > >
> > >> > > > > >> >> > > > I would love to see this FLIP-27 source interface
> > >> > > improvement
> > >> > > > > [1]
> > >> > > > > >> >> made
> > >> > > > > >> >> > to
> > >> > > > > >> >> > > > 1.11.3.
> > >> > > > > >> >> > > >
> > >> > > > > >> >> > > > [1]
> > https://issues.apache.org/jira/browse/FLINK-19698
> > >> > > > > >> >> > > >
> > >> > > > > >> >> > > > On Wed, Oct 28, 2020 at 12:32 AM Tzu-Li (Gordon)
> > Tai <
> > >> > > > > >> >> > > tzulitai@apache.org>
> > >> > > > > >> >> > > > wrote:
> > >> > > > > >> >> > > >
> > >> > > > > >> >> > > > > Thanks for the replies so far!
> > >> > > > > >> >> > > > >
> > >> > > > > >> >> > > > > Just to provide a brief update on the status of
> > >> > blockers
> > >> > > > for
> > >> > > > > >> >> 1.11.3
> > >> > > > > >> >> > so
> > >> > > > > >> >> > > > far:
> > >> > > > > >> >> > > > >
> > >> > > > > >> >> > > > >
> > >> > > > > >> >> > > > > *PR opened, pending reviewer*- [FLINK-19717]
> > >> > > > > >> >> > SourceReaderBase.pollNext
> > >> > > > > >> >> > > > may
> > >> > > > > >> >> > > > > return END_OF_INPUT if SplitReader.fetch throws
> (
> > >> > > > > >> >> > > > > https://github.com/apache/flink/pull/13776)
> > >> > > > > >> >> > > > >
> > >> > > > > >> >> > > > > *PR opened, reviewed + close to being merged*
> > >> > > > > >> >> > > > > - [FLINK-19741] Timer service should skip
> > restoring
> > >> > from
> > >> > > > raw
> > >> > > > > >> keyed
> > >> > > > > >> >> > > stream
> > >> > > > > >> >> > > > > if it isn't the writer (
> > >> > > > > >> >> https://github.com/apache/flink/pull/13761)
> > >> > > > > >> >> > > > > - [FLINK-19748] Raw keyed stream key group
> > iterator
> > >> > > should
> > >> > > > be
> > >> > > > > >> >> > skipping
> > >> > > > > >> >> > > > > unwritten key groups (
> > >> > > > > >> https://github.com/apache/flink/pull/13772)
> > >> > > > > >> >> > > > >
> > >> > > > > >> >> > > > > *Merged*
> > >> > > > > >> >> > > > > - [FLINK-19154] Application mode deletes HA data
> > in
> > >> > case
> > >> > > of
> > >> > > > > >> >> suspended
> > >> > > > > >> >> > > > > ZooKeeper connection
> > >> > > > > >> >> > > > > - [FLINK-19569] Upgrade ICU4J to 67.1+
> > >> > > > > >> >> > > > >
> > >> > > > > >> >> > > > > Right now as it seems, progress is mainly
> blocked
> > >> on a
> > >> > > > > reviewer
> > >> > > > > >> >> for
> > >> > > > > >> >> > > > > FLINK-19717.
> > >> > > > > >> >> > > > > Meanwhile, Xintong is keeping an eye on test
> > >> > > instabilities
> > >> > > > > [1]
> > >> > > > > >> to
> > >> > > > > >> >> see
> > >> > > > > >> >> > > if
> > >> > > > > >> >> > > > > there are any fixes that should be applied to
> > >> > > > `release-1.11`.
> > >> > > > > >> >> > > > >
> > >> > > > > >> >> > > > > This is also a reminder, that if there are other
> > >> > blockers
> > >> > > > > that
> > >> > > > > >> we
> > >> > > > > >> >> > need
> > >> > > > > >> >> > > to
> > >> > > > > >> >> > > > > be aware of, or a need to re-establish estimated
> > >> time
> > >> > for
> > >> > > > > >> getting
> > >> > > > > >> >> > fixes
> > >> > > > > >> >> > > > in
> > >> > > > > >> >> > > > > and delay the RC for 1.11.3, please do let us
> > know!
> > >> > > > > >> >> > > > >
> > >> > > > > >> >> > > > > Cheers,
> > >> > > > > >> >> > > > > Gordon
> > >> > > > > >> >> > > > >
> > >> > > > > >> >> > > > > [1]
> > >> > > > > >> >> > > > >
> > >> > > > > >> >> > > > >
> > >> > > > > >> >> > > >
> > >> > > > > >> >> > >
> > >> > > > > >> >> >
> > >> > > > > >> >>
> > >> > > > > >>
> > >> > > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> >
> https://issues.apache.org/jira/issues/?jql=project%20%3D%20FLINK%20AND%20status%20in%20(Open%2C%20%22In%20Progress%22%2C%20Reopened)%20AND%20affectedVersion%20in%20(1.11.0%2C%201.11.1%2C%201.11.2)%20AND%20labels%20%3D%20test-stability
> > >> > > > > >> >> > > > >
> > >> > > > > >> >> > > > > On Mon, Oct 26, 2020 at 9:43 PM Kostas Kloudas <
> > >> > > > > >> >> kkloudas@apache.org>
> > >> > > > > >> >> > > > > wrote:
> > >> > > > > >> >> > > > >
> > >> > > > > >> >> > > > > > +1 for releasing Flink 1.11.3 as it contains a
> > >> number
> > >> > > of
> > >> > > > > >> >> important
> > >> > > > > >> >> > > > > > fixes and thanks Gordon and Xintong for
> > >> volunteering.
> > >> > > > > >> >> > > > > >
> > >> > > > > >> >> > > > > > Cheers,
> > >> > > > > >> >> > > > > > Kostas
> > >> > > > > >> >> > > > > >
> > >> > > > > >> >> > > > > > On Mon, Oct 26, 2020 at 4:37 AM Yu Li <
> > >> > > carp84@gmail.com>
> > >> > > > > >> wrote:
> > >> > > > > >> >> > > > > > >
> > >> > > > > >> >> > > > > > > +1 for releasing Flink 1.11.3, and thanks
> > Gordon
> > >> > and
> > >> > > > > >> Xintong
> > >> > > > > >> >> for
> > >> > > > > >> >> > > > > > > volunteering as our release managers.
> > >> > > > > >> >> > > > > > >
> > >> > > > > >> >> > > > > > > Best Regards,
> > >> > > > > >> >> > > > > > > Yu
> > >> > > > > >> >> > > > > > >
> > >> > > > > >> >> > > > > > >
> > >> > > > > >> >> > > > > > > On Mon, 26 Oct 2020 at 09:45, Xintong Song <
> > >> > > > > >> >> > tonysong820@gmail.com>
> > >> > > > > >> >> > > > > > wrote:
> > >> > > > > >> >> > > > > > >
> > >> > > > > >> >> > > > > > > > Thanks Gordan for starting this
> discussion.
> > >> > > > > >> >> > > > > > > > My pleasure to help with the release
> > process.
> > >> > > > > >> >> > > > > > > >
> > >> > > > > >> >> > > > > > > > Thank you~
> > >> > > > > >> >> > > > > > > >
> > >> > > > > >> >> > > > > > > > Xintong Song
> > >> > > > > >> >> > > > > > > >
> > >> > > > > >> >> > > > > > > >
> > >> > > > > >> >> > > > > > > >
> > >> > > > > >> >> > > > > > > > On Fri, Oct 23, 2020 at 11:29 PM Till
> > >> Rohrmann <
> > >> > > > > >> >> > > > trohrmann@apache.org
> > >> > > > > >> >> > > > > >
> > >> > > > > >> >> > > > > > > > wrote:
> > >> > > > > >> >> > > > > > > >
> > >> > > > > >> >> > > > > > > > > Thanks for starting this discussion
> > Gordon.
> > >> > There
> > >> > > > are
> > >> > > > > >> over
> > >> > > > > >> >> > 100
> > >> > > > > >> >> > > > > issues
> > >> > > > > >> >> > > > > > > > > which are fixed for 1.11.3. Hence +1
> for a
> > >> > > soonish
> > >> > > > > >> 1.11.3
> > >> > > > > >> >> > > > release.
> > >> > > > > >> >> > > > > > Thanks
> > >> > > > > >> >> > > > > > > > > for volunteering as our release managers
> > >> Gordon
> > >> > > and
> > >> > > > > >> >> Xintong!
> > >> > > > > >> >> > > > > > > > >
> > >> > > > > >> >> > > > > > > > > Cheers,
> > >> > > > > >> >> > > > > > > > > Till
> > >> > > > > >> >> > > > > > > > >
> > >> > > > > >> >> > > > > > > > > On Fri, Oct 23, 2020 at 5:02 PM Tzu-Li
> > >> (Gordon)
> > >> > > > Tai <
> > >> > > > > >> >> > > > > > tzulitai@apache.org
> > >> > > > > >> >> > > > > > > > >
> > >> > > > > >> >> > > > > > > > > wrote:
> > >> > > > > >> >> > > > > > > > >
> > >> > > > > >> >> > > > > > > > >> Hi,
> > >> > > > > >> >> > > > > > > > >>
> > >> > > > > >> >> > > > > > > > >> Xintong and I would like to start a
> > >> discussion
> > >> > > for
> > >> > > > > >> >> releasing
> > >> > > > > >> >> > > > Flink
> > >> > > > > >> >> > > > > > > > 1.11.3
> > >> > > > > >> >> > > > > > > > >> soon.
> > >> > > > > >> >> > > > > > > > >>
> > >> > > > > >> >> > > > > > > > >> It seems like we already have a few
> > >> pressing
> > >> > > > issues
> > >> > > > > >> that
> > >> > > > > >> >> > needs
> > >> > > > > >> >> > > > to
> > >> > > > > >> >> > > > > be
> > >> > > > > >> >> > > > > > > > >> included in a new hotfix release:
> > >> > > > > >> >> > > > > > > > >>
> > >> > > > > >> >> > > > > > > > >>    - Heap-based timers’ restore
> behaviour
> > >> is
> > >> > > > > causing a
> > >> > > > > >> >> > > critical
> > >> > > > > >> >> > > > > > recovery
> > >> > > > > >> >> > > > > > > > >>    issue for StateFun [1] [2] [3].
> > >> > > > > >> >> > > > > > > > >>    - There are several robustness
> issues
> > >> for
> > >> > the
> > >> > > > > >> FLIP-27
> > >> > > > > >> >> new
> > >> > > > > >> >> > > > > source
> > >> > > > > >> >> > > > > > API,
> > >> > > > > >> >> > > > > > > > >>    such as [4]. We already have some
> > users
> > >> > using
> > >> > > > the
> > >> > > > > >> >> FLIP-27
> > >> > > > > >> >> > > API
> > >> > > > > >> >> > > > > > with
> > >> > > > > >> >> > > > > > > > >> 1.11.x,
> > >> > > > > >> >> > > > > > > > >>    so it would be important to get
> those
> > >> fixes
> > >> > > in
> > >> > > > > for
> > >> > > > > >> >> 1.11.x
> > >> > > > > >> >> > > as
> > >> > > > > >> >> > > > > > well.
> > >> > > > > >> >> > > > > > > > >>
> > >> > > > > >> >> > > > > > > > >> Apart from the issues that are already
> > >> marked
> > >> > as
> > >> > > > > >> blocker
> > >> > > > > >> >> for
> > >> > > > > >> >> > > > > 1.11.3
> > >> > > > > >> >> > > > > > in
> > >> > > > > >> >> > > > > > > > our
> > >> > > > > >> >> > > > > > > > >> JIRA [5], please let us know in this
> > >> thread if
> > >> > > > there
> > >> > > > > >> is
> > >> > > > > >> >> > > already
> > >> > > > > >> >> > > > > > ongoing
> > >> > > > > >> >> > > > > > > > >> work for other important fixes that we
> > >> should
> > >> > > try
> > >> > > > to
> > >> > > > > >> >> > include.
> > >> > > > > >> >> > > > > > > > >>
> > >> > > > > >> >> > > > > > > > >> Xintong and I would like to volunteer
> for
> > >> > > managing
> > >> > > > > >> this
> > >> > > > > >> >> > > release,
> > >> > > > > >> >> > > > > and
> > >> > > > > >> >> > > > > > > > will
> > >> > > > > >> >> > > > > > > > >> try to communicate the priority of
> > pending
> > >> > > > blockers
> > >> > > > > >> over
> > >> > > > > >> >> the
> > >> > > > > >> >> > > > next
> > >> > > > > >> >> > > > > > few
> > >> > > > > >> >> > > > > > > > >> days.
> > >> > > > > >> >> > > > > > > > >> Since the aforementioned issues are
> quite
> > >> > > > critical,
> > >> > > > > >> we’d
> > >> > > > > >> >> > like
> > >> > > > > >> >> > > to
> > >> > > > > >> >> > > > > aim
> > >> > > > > >> >> > > > > > > > >> for a *feature
> > >> > > > > >> >> > > > > > > > >> freeze by the end of next week (Oct.
> > 30th)*
> > >> > and
> > >> > > > > start
> > >> > > > > >> the
> > >> > > > > >> >> > > > release
> > >> > > > > >> >> > > > > > voting
> > >> > > > > >> >> > > > > > > > >> process the week after.
> > >> > > > > >> >> > > > > > > > >> If that is too short of a notice and
> you
> > >> might
> > >> > > > need
> > >> > > > > >> more
> > >> > > > > >> >> > time,
> > >> > > > > >> >> > > > > > please
> > >> > > > > >> >> > > > > > > > let
> > >> > > > > >> >> > > > > > > > >> us know!
> > >> > > > > >> >> > > > > > > > >>
> > >> > > > > >> >> > > > > > > > >> Cheers,
> > >> > > > > >> >> > > > > > > > >> Gordon
> > >> > > > > >> >> > > > > > > > >>
> > >> > > > > >> >> > > > > > > > >> [1]
> > >> > > > > https://issues.apache.org/jira/browse/FLINK-19692
> > >> > > > > >> >> > > > > > > > >> [2]
> > >> > > > > https://issues.apache.org/jira/browse/FLINK-19741
> > >> > > > > >> >> > > > > > > > >> [3]
> > >> > > > > https://issues.apache.org/jira/browse/FLINK-19748
> > >> > > > > >> >> > > > > > > > >> [4]
> > >> > > > > https://issues.apache.org/jira/browse/FLINK-19717
> > >> > > > > >> >> > > > > > > > >> [5]
> > >> > > > > >> >> > > > > > > > >>
> > >> > > > > >> >> > > > > > > > >>
> > >> > > > > >> >> > > > > > > >
> > >> > > > > >> >> > > > > >
> > >> > > > > >> >> > > > >
> > >> > > > > >> >> > > >
> > >> > > > > >> >> > >
> > >> > > > > >> >> >
> > >> > > > > >> >>
> > >> > > > > >>
> > >> > > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> >
> https://issues.apache.org/jira/issues/?jql=project%20%3D%20FLINK%20AND%20priority%20%3D%20Blocker%20AND%20fixVersion%20%3D%201.11.3
> > >> > > > > >> >> > > > > > > > >>
> > >> > > > > >> >> > > > > > > > >
> > >> > > > > >> >> > > > > > > >
> > >> > > > > >> >> > > > > >
> > >> > > > > >> >> > > > >
> > >> > > > > >> >> > > >
> > >> > > > > >> >> > >
> > >> > > > > >> >> >
> > >> > > > > >> >> >
> > >> > > > > >> >> > --
> > >> > > > > >> >> > Best, Jingsong Lee
> > >> > > > > >> >> >
> > >> > > > > >> >>
> > >> > > > > >> >
> > >> > > > > >>
> > >> > > > > >
> > >> > > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> > >
> >
>

Re: [DISCUSS] Releasing Apache Flink 1.11.3

Posted by Xintong Song <to...@gmail.com>.
Hi devs,

I'd like to provide an update on the progress of preparing release 1.11.3.

*Blockers*
We currently have 3 remaining blockers. (3 resolved and 1 emerged compared
to last week)

- [FLINK-19698] Add close() method and onCheckpointComplete() to the Source.
The issue has been fixed on the master branch. It's currently blocked on
the FLIP-27 backportings to backport it to the 1.11 branch.

- [FLINK-19717] SourceReaderBase.pollNext may return END_OF_INPUT if
SplitReader.fetch throws
A PR has been opened and reviewed. From the discussions on the PR, it looks
close to mergeable.

- [FLINK-19816] Flink restored from a wrong checkpoint (a very old one and
not the last completed one)
This is a newly emerged blocker and Matthias is working on it.

*Test Instabilities*
We currently have 27 test instabilities[1].
AFAIK, none of them are as serious as to block the 1.11.3 release.

*FLIP-27 Backprotings*

I noticed that there's no jira issues opened on the FLIP-27 backporting
efforts, which is part of the major efforts planned for the 1.11.3 release,
making it hard to track the progress.


@Stephan and @Becket, could you please share the updates on the backporting
efforts? How is the progress and when are the efforts expected to be
finished? It would be appreciated and helpful if we can have a jira ticket
to track the backporing progress publicly.


Thank you~

Xintong Song


[1]
https://issues.apache.org/jira/browse/FLINK-16908?filter=12348580&jql=project%20%3D%20FLINK%20AND%20resolution%20%3D%20Unresolved%20AND%20affectedVersion%20in%20(1.11.0%2C%201.11.1%2C%201.11.2%2C%201.11.3)%20AND%20labels%20%3D%20test-stability%20ORDER%20BY%20updated%20DESC%2C%20created%20DESC

On Wed, Nov 4, 2020 at 3:58 PM Tzu-Li (Gordon) Tai <tz...@apache.org>
wrote:

> Thanks for the efforts so far with pushing for 1.11.3.
>
> I'd like to provide a mid-week update on how we're looking with all the
> pending blockers and backports:
>
> *Blockers**:*
>
> Currently all blockers either have an open PR, or have been merged. I'll
> highlight below the blockers *that still require reviewing efforts for them
> to move forward:*
>
> - [FLINK-19909] Flink application in attach mode could not terminate when
> the only job is canceled. PR: https://github.com/apache/flink/pull/13911
> - [FLINK-19717] SourceReaderBase.pollNext may return END_OF_INPUT if
> SplitReader.fetch throws. PR: https://github.com/apache/flink/pull/13776
>
> The above PRs currently have no reviews at all yet. They seem to already
> have designated reviewers.
>
>
> *Backports:*
> - FLIP-27 robustness improvement backports: Stephan is currently working on
> backporting several FLIP-27 changes. There are no PRs yet for the
> backported changes.
> - DataStreamUtils.collect() refactorings backport: Steven Wu mentioned to
> backport this, but AFAIK this isn't assigned to anyone yet.
>
> *ETAs*:
>
> I'd like to request ETAs for the remaining backports, to prevent a creep in
> the scope of this bugfix release.
>
> We already have Flink users that would benefit from fixes that have been
> merged to release-1.11 already, so technically speaking the backports
> should be considered "nice-to-have" (to the best of my knowledge of the
> changes) and could potentially be moved to a follow-up 1.11.4.
> Most notably, the Stateful Functions project is already waiting on Flink
> 1.11.3 to address critical recovery issues (please see the StateFun 2.2.1
> release discussion thread [1]).
>
> @Stephan Ewen <se...@apache.org> @Becket Qin <be...@gmail.com> could
> you provide an ETA for the FLIP-27 backports? It would help to get a better
> estimate to decide how we proceed here.
>
> Cheers,
> Gordon
>
> [1]
>
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Releasing-StateFun-hotfix-version-2-2-1-td46239.html
>
> On Wed, Nov 4, 2020 at 3:16 PM Tzu-Li (Gordon) Tai <tz...@apache.org>
> wrote:
>
> > > The collect() utils can be picked back, I see no issue with that (it is
> > isolated utilities).
> >
> > Just checking on all the requested backports mentioned in this thread,
> and
> > figuring out which ones seem to still be unassigned / open.
> >
> > Is someone working on backporting
> >
> https://github.com/apache/flink/commit/09a7a66b7313fea64817fe960a8da1265b428efc
> > to release-1.11 at the moment?
> >
> > On Wed, Nov 4, 2020 at 1:39 AM Steven Wu <st...@gmail.com> wrote:
> >
> >> @Stephan Ewen <se...@apache.org> yeah, we can do that. don't worry
> about
> >> it. your earlier email had the perfect explanation on why file source
> >> shouldn't be backported.
> >>
> >> On Tue, Nov 3, 2020 at 3:37 AM Stephan Ewen <se...@apache.org> wrote:
> >>
> >> > @Steven would it be possible to initially copy some of the code into
> the
> >> > iceberg source and later replace it by a dependency on the Flink file
> >> > source?
> >> >
> >> > On Mon, Nov 2, 2020 at 8:33 PM Steven Wu <st...@gmail.com>
> wrote:
> >> >
> >> > > Stephan, thanks a lot for explaining the file connector. that makes
> >> > sense.
> >> > >
> >> > > I was asking because we were trying to reuse some of the
> >> implementations
> >> > in
> >> > > the file source for Iceberg source. Flink Iceberg source lives in
> the
> >> > > Iceberg repo, which is not possible to code against the master
> branch
> >> of
> >> > > the Flink code.
> >> > >
> >> > > On Mon, Nov 2, 2020 at 3:31 AM Stephan Ewen <se...@apache.org>
> wrote:
> >> > >
> >> > > > Hi Steven!
> >> > > >
> >> > > > So far there are no plans to pick back the file system connector
> >> code.
> >> > > This
> >> > > > is still evolving and not finalized for 1.12, so I don't feel it
> is
> >> a
> >> > > good
> >> > > > candidate to be backported.
> >> > > > However, with the base connector changes backported, you should be
> >> able
> >> > > to
> >> > > > run the file connector code from master against 1.11.3.
> >> > > >
> >> > > > The collect() utils can be picked back, I see no issue with that
> >> (it is
> >> > > > isolated utilities).
> >> > > >
> >> > > > Best,
> >> > > > Stephan
> >> > > >
> >> > > >
> >> > > > On Mon, Nov 2, 2020 at 3:02 AM Steven Wu <st...@gmail.com>
> >> wrote:
> >> > > >
> >> > > > > Basically, it would be great to get the latest code in the
> >> > > > > flink-connector-files (FLIP-27).
> >> > > > >
> >> > > > > On Sat, Oct 31, 2020 at 9:57 AM Steven Wu <stevenz3wu@gmail.com
> >
> >> > > wrote:
> >> > > > >
> >> > > > > > Stephan, it will be great if we can also backport the
> >> > DataStreamUtils
> >> > > > > > related commits that help with collecting output from
> unbounded
> >> > > > streams.
> >> > > > > > e.g.
> >> > > > > >
> >> > > > > >
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
> https://github.com/apache/flink/commit/09a7a66b7313fea64817fe960a8da1265b428efc
> >> > > > > >
> >> > > > > > I tried to copy and paste the code to unblock myself. but it
> >> > quickly
> >> > > > got
> >> > > > > > into the rabbit hole of more and more code.
> >> > > > > >
> >> > > > > > On Fri, Oct 30, 2020 at 11:02 AM Stephan Ewen <
> sewen@apache.org
> >> >
> >> > > > wrote:
> >> > > > > >
> >> > > > > >> I have started with backporting the source API changes. Some
> >> minor
> >> > > > > >> conflicts to solve, will need a bit more to finish this.
> >> > > > > >>
> >> > > > > >> On Fri, Oct 30, 2020 at 7:25 AM Tzu-Li (Gordon) Tai <
> >> > > > > tzulitai@apache.org>
> >> > > > > >> wrote:
> >> > > > > >>
> >> > > > > >> > @Stephan Ewen <se...@apache.org>
> >> > > > > >> > Are there already plans or ongoing efforts for backporting
> >> the
> >> > > list
> >> > > > of
> >> > > > > >> > FLIP-27 changes that you posted?
> >> > > > > >> >
> >> > > > > >> > On Thu, Oct 29, 2020 at 7:08 PM Xintong Song <
> >> > > tonysong820@gmail.com
> >> > > > >
> >> > > > > >> > wrote:
> >> > > > > >> >
> >> > > > > >> >> Hi folks,
> >> > > > > >> >>
> >> > > > > >> >> Just to provide some updates concerning the status on the
> >> > > > > >> >> test instabilities.
> >> > > > > >> >>
> >> > > > > >> >> Currently, we have 30 unresolved tickets labeled with
> >> `Affects
> >> > > > > Version`
> >> > > > > >> >> 1.11.x.
> >> > > > > >> >>
> >> > > > > >> >>
> >> > > > > >>
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
> https://issues.apache.org/jira/browse/FLINK-19775?filter=12348580&jql=project%20%3D%20FLINK%20AND%20resolution%20%3D%20Unresolved%20AND%20affectedVersion%20in%20(1.11.0%2C%201.11.1%2C%201.11.2%2C%201.11.3)%20AND%20labels%20%3D%20test-stability%20ORDER%20BY%20created%20DESC
> >> > > > > >> >>
> >> > > > > >> >> Among the 30 tickets, 11 of them are:
> >> > > > > >> >> - Have occured in the recent 3 months
> >> > > > > >> >> - Not confirmed to be pure testability issues
> >> > > > > >> >> - Not confirmed to be rare condition cases
> >> > > > > >> >>
> >> > > > > >> >> It would be nice if someone familiar with these components
> >> can
> >> > > > take a
> >> > > > > >> look
> >> > > > > >> >> into these issues.
> >> > > > > >> >>
> >> > > > > >> >> - https://issues.apache.org/jira/browse/FLINK-17159 (ES6)
> >> > > > > >> >> - https://issues.apache.org/jira/browse/FLINK-17912
> (Kafka)
> >> > > > > >> >> - https://issues.apache.org/jira/browse/FLINK-17949
> (Kafka)
> >> > > > > >> >> ⁃ https://issues.apache.org/jira/browse/FLINK-18444
> (Kafka)
> >> > > > > >> >> - https://issues.apache.org/jira/browse/FLINK-18634
> (Kafka)
> >> > > > > >> >> - https://issues.apache.org/jira/browse/FLINK-18648
> (Kafka)
> >> > > > > >> >> - https://issues.apache.org/jira/browse/FLINK-18807
> (Kafka)
> >> > > > > >> >> - https://issues.apache.org/jira/browse/FLINK-19369
> >> > > > (BlobClientTest)
> >> > > > > >> >> - https://issues.apache.org/jira/browse/FLINK-19436
> (TPCDS)
> >> > > > > >> >> - https://issues.apache.org/jira/browse/FLINK-19690
> >> > > > (Format/Parquet)
> >> > > > > >> >> - https://issues.apache.org/jira/browse/FLINK-19775
> >> > > > > >> >> (SystemProcessingTimeServiceTest)
> >> > > > > >> >>
> >> > > > > >> >> Thank you~
> >> > > > > >> >>
> >> > > > > >> >> Xintong Song
> >> > > > > >> >>
> >> > > > > >> >>
> >> > > > > >> >>
> >> > > > > >> >> On Thu, Oct 29, 2020 at 10:21 AM Jingsong Li <
> >> > > > jingsonglee0@gmail.com
> >> > > > > >
> >> > > > > >> >> wrote:
> >> > > > > >> >>
> >> > > > > >> >> > +1 to backport the FLIP-27 adjustments to 1.11.x.
> >> > > > > >> >> >
> >> > > > > >> >> > If possible, that would be great. Many people are
> looking
> >> > > forward
> >> > > > > to
> >> > > > > >> the
> >> > > > > >> >> > FLIP-27 interface, but they don't want to take the risk
> to
> >> > > > upgrade
> >> > > > > to
> >> > > > > >> >> 1.12
> >> > > > > >> >> > (And wait 1.12). After all, 1.11 is a relatively stable
> >> > > version.
> >> > > > > >> >> >
> >> > > > > >> >> > Best,
> >> > > > > >> >> > Jingsong
> >> > > > > >> >> >
> >> > > > > >> >> > On Thu, Oct 29, 2020 at 1:24 AM Stephan Ewen <
> >> > sewen@apache.org
> >> > > >
> >> > > > > >> wrote:
> >> > > > > >> >> >
> >> > > > > >> >> > > Thanks for starting this.
> >> > > > > >> >> > >
> >> > > > > >> >> > > +1 form my side to backport the FLIP-27 adjustments to
> >> > > 1.11.x.
> >> > > > > >> >> > >
> >> > > > > >> >> > > There were quite a few changes, and I think we need to
> >> > > > > cherry-pick
> >> > > > > >> >> them
> >> > > > > >> >> > all
> >> > > > > >> >> > > to not get some inconsistent mix of changes and many
> >> merge
> >> > > > > >> conflicts.
> >> > > > > >> >> > > I made a list below of what we need to add to
> >> > "release-1.11".
> >> > > > > >> >> > >
> >> > > > > >> >> > > *#### Core Source API Changes to backport (in REVERSE
> >> > order)*
> >> > > > > >> >> > >
> >> > > > > >> >> > >   (Use: "git log
> >> > > > > >> >> > >
> >> > > > flink-core/src/main/java/org/apache/flink/api/connector/source")
> >> > > > > >> >> > >
> >> > > > > >> >> > > commit 162c072e9265a7b6dd9d6f5459eb7974091c4c4e
> >> > > > > >> >> > > [FLINK-19492][core] Consolidate Source Events between
> >> > Source
> >> > > > API
> >> > > > > >> and
> >> > > > > >> >> > Split
> >> > > > > >> >> > > Reader API
> >> > > > > >> >> > >
> >> > > > > >> >> > > commit 428fbbad85379f71a9fbf1a1ab3e3dcfa9006436
> >> > > > > >> >> > > [FLINK-19205][core] Add access to configuration and
> >> > hostname
> >> > > in
> >> > > > > the
> >> > > > > >> >> > > SourceReaderContext
> >> > > > > >> >> > >
> >> > > > > >> >> > > commit 723e1790744ddc78d19c7c978442af1383f38d33
> >> > > > > >> >> > > [hotfix][core] Add to Source Enumerator convenience
> >> methods
> >> > > to
> >> > > > > >> assign
> >> > > > > >> >> > > single split
> >> > > > > >> >> > >
> >> > > > > >> >> > > *#### Connector Base Changes to Backport (in REVERSE
> >> > order)*
> >> > > > > >> >> > >
> >> > > > > >> >> > >   (Use: "git log
> flink-connectors/flink-connector-base")
> >> > > > > >> >> > >
> >> > > > > >> >> > > commit 401f56fe9d6b0271260edf9787cdcbfe4d03874d
> >> > > > > >> >> > > [FLINK-19427][FLINK-19489][tests] Fix test conditions
> >> for
> >> > > > > >> >> > >
> 'SplitFetcherTest.testNotifiesWhenGoingIdleConcurrent()'
> >> > > > > >> >> > >
> >> > > > > >> >> > > commit 68c5c2ff779d82a1ff81ffaf60d8a1b283797db1
> >> > > > > >> >> > > [FLINK-19448][connector base] Explicitly check for
> >> > > un-expected
> >> > > > > >> >> condition
> >> > > > > >> >> > > that would leave an inconsistent state
> >> > > > > >> >> > >
> >> > > > > >> >> > > commit 162c072e9265a7b6dd9d6f5459eb7974091c4c4e
> >> > > > > >> >> > > [FLINK-19492][core] Consolidate Source Events between
> >> > Source
> >> > > > API
> >> > > > > >> and
> >> > > > > >> >> > Split
> >> > > > > >> >> > > Reader API
> >> > > > > >> >> > >
> >> > > > > >> >> > > commit c1ca7a4c7c21ec8868c14cdddd43c559625b794c
> >> > > > > >> >> > > [refactor][tests] Move some source test utils from
> >> > > > > >> >> flink-connector-base
> >> > > > > >> >> > to
> >> > > > > >> >> > > flink-core
> >> > > > > >> >> > >
> >> > > > > >> >> > > commit ee5c4c211c35c70d28252363bbc8400453609977
> >> > > > > >> >> > > [FLINK-19251][connectors] Avoid confusing queue
> >> handling in
> >> > > > > >> >> > > "SplitReader.handleSplitsChanges()"
> >> > > > > >> >> > >
> >> > > > > >> >> > > commit 5abef56b2bf85bcac786f6b16b6899b6cced7176
> >> > > > > >> >> > > [FLINK-19250][connectors] Fix error propagation in
> >> > connector
> >> > > > base
> >> > > > > >> >> > > (SplitFetcherManager).
> >> > > > > >> >> > >
> >> > > > > >> >> > > commit 8fcca837c55a9216595ee4c03038b52747098dbb
> >> > > > > >> >> > > [hotfix][connectors] Improve JavaDocs for
> >> > > > > >> SingleThreadFetcherManager
> >> > > > > >> >> > >
> >> > > > > >> >> > > commit 4700bb5dde3303cbe98882f6beb7379425717b01
> >> > > > > >> >> > > [FLINK-19225][connectors] Various small improvements
> to
> >> > > > > >> >> SourceReaderBase
> >> > > > > >> >> > > (part 2)
> >> > > > > >> >> > >
> >> > > > > >> >> > > commit 12261c6b7ed6478a9b9f6a69cb58246b83cab9b7
> >> > > > > >> >> > > [FLINK-17393][connectors] (follow-up) Wakeup the
> >> > > SplitFetchers
> >> > > > > more
> >> > > > > >> >> > > elegantly.
> >> > > > > >> >> > >
> >> > > > > >> >> > > commit c60aaff0249bfd6b5871b7f82e03efc487a54d6b
> >> > > > > >> >> > > [hotfix][tests] Extend test coverage for
> >> > > > > >> FutureCompletingBlockingQueue
> >> > > > > >> >> > >
> >> > > > > >> >> > > commit cef8a587d7fd2fe64cc644da5ed095d82e46f631
> >> > > > > >> >> > > [FLINK-19245][connectors] Set default capacity for
> >> > > > > >> >> > > FutureCompletingBlockingQueue.
> >> > > > > >> >> > >
> >> > > > > >> >> > > commit 4ea95782b4c6a2538153d4d16ad3f4839c7de0fb
> >> > > > > >> >> > > [FLINK-19223][connectors] Simplify Availability Future
> >> > Model
> >> > > in
> >> > > > > >> Base
> >> > > > > >> >> > > Connector
> >> > > > > >> >> > >
> >> > > > > >> >> > > commit 511857049ba30c8ff0ee56da551fa4a479dc583e
> >> > > > > >> >> > > [FLINK-18128][connectors] Ensure idle split fetchers
> >> lead
> >> > to
> >> > > > > >> >> availability
> >> > > > > >> >> > > notifications.
> >> > > > > >> >> > >
> >> > > > > >> >> > > commit a8206467af0830dcb89623ea068b5ca3b3450c92
> >> > > > > >> >> > > [refactor][core] Eagerly initialize the FetchTask to
> >> > support
> >> > > > > proper
> >> > > > > >> >> unit
> >> > > > > >> >> > > testing
> >> > > > > >> >> > >
> >> > > > > >> >> > > commit 3b2f54bcb437f98e6137c904045cc51072b5c06b
> >> > > > > >> >> > > [hotfix][tests] Move constants in SplitFetcherTest
> >> relevant
> >> > > to
> >> > > > > only
> >> > > > > >> >> one
> >> > > > > >> >> > > test into test method
> >> > > > > >> >> > >
> >> > > > > >> >> > > commit d7625760a75a508bf05bcddc380bb4d62ee1743e
> >> > > > > >> >> > > [FLINK-19225][connectors] Various small improvements
> to
> >> > > > > >> >> SourceReaderBase
> >> > > > > >> >> > >
> >> > > > > >> >> > > commit a5b0d3297748c1be47ad579a88f24df2255a8df1
> >> > > > > >> >> > > [FLINK-17393][connectors] Wakeup the SplitFetchers
> more
> >> > > > > elegantly.
> >> > > > > >> >> > >
> >> > > > > >> >> > > commit f42a3ebc3e81a034b7221a803c153636fef34903
> >> > > > > >> >> > > [FLINK-18680][connectors] Make connector base
> >> > > > RecordsWithSplitIds
> >> > > > > >> more
> >> > > > > >> >> > > lightweight.
> >> > > > > >> >> > >
> >> > > > > >> >> > > commit e3d273de822b085183d09b275a445879ff94b350
> >> > > > > >> >> > > [FLINK-19162][connectors] Add 'recycle()' to the
> >> > > > > >> RecordsWithSplitIds
> >> > > > > >> >> to
> >> > > > > >> >> > > support reuse of heavy objects.
> >> > > > > >> >> > >
> >> > > > > >> >> > > commit 8ebc464c2520453a70001cd712abc8dee6ee89e0
> >> > > > > >> >> > > [hotfix][testing] Add a set of parameterizable testing
> >> > mocks
> >> > > > for
> >> > > > > >> the
> >> > > > > >> >> > Split
> >> > > > > >> >> > > Reader API
> >> > > > > >> >> > >
> >> > > > > >> >> > > commit 930a07438be1185388d7150640f294dfe2a5d378
> >> > > > > >> >> > > [hotfix][connectors] Add RequestSplitEvent to
> >> > > > > >> 'flink-connector-base'
> >> > > > > >> >> > >
> >> > > > > >> >> > > commit e945ce8a933bc378844782f784ca473c767ca159
> >> > > > > >> >> > > [refactor][connectors] Backport of the connector-base
> >> > > exception
> >> > > > > >> >> handling
> >> > > > > >> >> > > from the Kafka Connector Pull Request
> >> > > > > >> >> > >
> >> > > > > >> >> > >
> >> > > > > >> >> > > On Wed, Oct 28, 2020 at 5:49 PM Steven Wu <
> >> > > > stevenz3wu@gmail.com>
> >> > > > > >> >> wrote:
> >> > > > > >> >> > >
> >> > > > > >> >> > > > I would love to see this FLIP-27 source interface
> >> > > improvement
> >> > > > > [1]
> >> > > > > >> >> made
> >> > > > > >> >> > to
> >> > > > > >> >> > > > 1.11.3.
> >> > > > > >> >> > > >
> >> > > > > >> >> > > > [1]
> https://issues.apache.org/jira/browse/FLINK-19698
> >> > > > > >> >> > > >
> >> > > > > >> >> > > > On Wed, Oct 28, 2020 at 12:32 AM Tzu-Li (Gordon)
> Tai <
> >> > > > > >> >> > > tzulitai@apache.org>
> >> > > > > >> >> > > > wrote:
> >> > > > > >> >> > > >
> >> > > > > >> >> > > > > Thanks for the replies so far!
> >> > > > > >> >> > > > >
> >> > > > > >> >> > > > > Just to provide a brief update on the status of
> >> > blockers
> >> > > > for
> >> > > > > >> >> 1.11.3
> >> > > > > >> >> > so
> >> > > > > >> >> > > > far:
> >> > > > > >> >> > > > >
> >> > > > > >> >> > > > >
> >> > > > > >> >> > > > > *PR opened, pending reviewer*- [FLINK-19717]
> >> > > > > >> >> > SourceReaderBase.pollNext
> >> > > > > >> >> > > > may
> >> > > > > >> >> > > > > return END_OF_INPUT if SplitReader.fetch throws (
> >> > > > > >> >> > > > > https://github.com/apache/flink/pull/13776)
> >> > > > > >> >> > > > >
> >> > > > > >> >> > > > > *PR opened, reviewed + close to being merged*
> >> > > > > >> >> > > > > - [FLINK-19741] Timer service should skip
> restoring
> >> > from
> >> > > > raw
> >> > > > > >> keyed
> >> > > > > >> >> > > stream
> >> > > > > >> >> > > > > if it isn't the writer (
> >> > > > > >> >> https://github.com/apache/flink/pull/13761)
> >> > > > > >> >> > > > > - [FLINK-19748] Raw keyed stream key group
> iterator
> >> > > should
> >> > > > be
> >> > > > > >> >> > skipping
> >> > > > > >> >> > > > > unwritten key groups (
> >> > > > > >> https://github.com/apache/flink/pull/13772)
> >> > > > > >> >> > > > >
> >> > > > > >> >> > > > > *Merged*
> >> > > > > >> >> > > > > - [FLINK-19154] Application mode deletes HA data
> in
> >> > case
> >> > > of
> >> > > > > >> >> suspended
> >> > > > > >> >> > > > > ZooKeeper connection
> >> > > > > >> >> > > > > - [FLINK-19569] Upgrade ICU4J to 67.1+
> >> > > > > >> >> > > > >
> >> > > > > >> >> > > > > Right now as it seems, progress is mainly blocked
> >> on a
> >> > > > > reviewer
> >> > > > > >> >> for
> >> > > > > >> >> > > > > FLINK-19717.
> >> > > > > >> >> > > > > Meanwhile, Xintong is keeping an eye on test
> >> > > instabilities
> >> > > > > [1]
> >> > > > > >> to
> >> > > > > >> >> see
> >> > > > > >> >> > > if
> >> > > > > >> >> > > > > there are any fixes that should be applied to
> >> > > > `release-1.11`.
> >> > > > > >> >> > > > >
> >> > > > > >> >> > > > > This is also a reminder, that if there are other
> >> > blockers
> >> > > > > that
> >> > > > > >> we
> >> > > > > >> >> > need
> >> > > > > >> >> > > to
> >> > > > > >> >> > > > > be aware of, or a need to re-establish estimated
> >> time
> >> > for
> >> > > > > >> getting
> >> > > > > >> >> > fixes
> >> > > > > >> >> > > > in
> >> > > > > >> >> > > > > and delay the RC for 1.11.3, please do let us
> know!
> >> > > > > >> >> > > > >
> >> > > > > >> >> > > > > Cheers,
> >> > > > > >> >> > > > > Gordon
> >> > > > > >> >> > > > >
> >> > > > > >> >> > > > > [1]
> >> > > > > >> >> > > > >
> >> > > > > >> >> > > > >
> >> > > > > >> >> > > >
> >> > > > > >> >> > >
> >> > > > > >> >> >
> >> > > > > >> >>
> >> > > > > >>
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
> https://issues.apache.org/jira/issues/?jql=project%20%3D%20FLINK%20AND%20status%20in%20(Open%2C%20%22In%20Progress%22%2C%20Reopened)%20AND%20affectedVersion%20in%20(1.11.0%2C%201.11.1%2C%201.11.2)%20AND%20labels%20%3D%20test-stability
> >> > > > > >> >> > > > >
> >> > > > > >> >> > > > > On Mon, Oct 26, 2020 at 9:43 PM Kostas Kloudas <
> >> > > > > >> >> kkloudas@apache.org>
> >> > > > > >> >> > > > > wrote:
> >> > > > > >> >> > > > >
> >> > > > > >> >> > > > > > +1 for releasing Flink 1.11.3 as it contains a
> >> number
> >> > > of
> >> > > > > >> >> important
> >> > > > > >> >> > > > > > fixes and thanks Gordon and Xintong for
> >> volunteering.
> >> > > > > >> >> > > > > >
> >> > > > > >> >> > > > > > Cheers,
> >> > > > > >> >> > > > > > Kostas
> >> > > > > >> >> > > > > >
> >> > > > > >> >> > > > > > On Mon, Oct 26, 2020 at 4:37 AM Yu Li <
> >> > > carp84@gmail.com>
> >> > > > > >> wrote:
> >> > > > > >> >> > > > > > >
> >> > > > > >> >> > > > > > > +1 for releasing Flink 1.11.3, and thanks
> Gordon
> >> > and
> >> > > > > >> Xintong
> >> > > > > >> >> for
> >> > > > > >> >> > > > > > > volunteering as our release managers.
> >> > > > > >> >> > > > > > >
> >> > > > > >> >> > > > > > > Best Regards,
> >> > > > > >> >> > > > > > > Yu
> >> > > > > >> >> > > > > > >
> >> > > > > >> >> > > > > > >
> >> > > > > >> >> > > > > > > On Mon, 26 Oct 2020 at 09:45, Xintong Song <
> >> > > > > >> >> > tonysong820@gmail.com>
> >> > > > > >> >> > > > > > wrote:
> >> > > > > >> >> > > > > > >
> >> > > > > >> >> > > > > > > > Thanks Gordan for starting this discussion.
> >> > > > > >> >> > > > > > > > My pleasure to help with the release
> process.
> >> > > > > >> >> > > > > > > >
> >> > > > > >> >> > > > > > > > Thank you~
> >> > > > > >> >> > > > > > > >
> >> > > > > >> >> > > > > > > > Xintong Song
> >> > > > > >> >> > > > > > > >
> >> > > > > >> >> > > > > > > >
> >> > > > > >> >> > > > > > > >
> >> > > > > >> >> > > > > > > > On Fri, Oct 23, 2020 at 11:29 PM Till
> >> Rohrmann <
> >> > > > > >> >> > > > trohrmann@apache.org
> >> > > > > >> >> > > > > >
> >> > > > > >> >> > > > > > > > wrote:
> >> > > > > >> >> > > > > > > >
> >> > > > > >> >> > > > > > > > > Thanks for starting this discussion
> Gordon.
> >> > There
> >> > > > are
> >> > > > > >> over
> >> > > > > >> >> > 100
> >> > > > > >> >> > > > > issues
> >> > > > > >> >> > > > > > > > > which are fixed for 1.11.3. Hence +1 for a
> >> > > soonish
> >> > > > > >> 1.11.3
> >> > > > > >> >> > > > release.
> >> > > > > >> >> > > > > > Thanks
> >> > > > > >> >> > > > > > > > > for volunteering as our release managers
> >> Gordon
> >> > > and
> >> > > > > >> >> Xintong!
> >> > > > > >> >> > > > > > > > >
> >> > > > > >> >> > > > > > > > > Cheers,
> >> > > > > >> >> > > > > > > > > Till
> >> > > > > >> >> > > > > > > > >
> >> > > > > >> >> > > > > > > > > On Fri, Oct 23, 2020 at 5:02 PM Tzu-Li
> >> (Gordon)
> >> > > > Tai <
> >> > > > > >> >> > > > > > tzulitai@apache.org
> >> > > > > >> >> > > > > > > > >
> >> > > > > >> >> > > > > > > > > wrote:
> >> > > > > >> >> > > > > > > > >
> >> > > > > >> >> > > > > > > > >> Hi,
> >> > > > > >> >> > > > > > > > >>
> >> > > > > >> >> > > > > > > > >> Xintong and I would like to start a
> >> discussion
> >> > > for
> >> > > > > >> >> releasing
> >> > > > > >> >> > > > Flink
> >> > > > > >> >> > > > > > > > 1.11.3
> >> > > > > >> >> > > > > > > > >> soon.
> >> > > > > >> >> > > > > > > > >>
> >> > > > > >> >> > > > > > > > >> It seems like we already have a few
> >> pressing
> >> > > > issues
> >> > > > > >> that
> >> > > > > >> >> > needs
> >> > > > > >> >> > > > to
> >> > > > > >> >> > > > > be
> >> > > > > >> >> > > > > > > > >> included in a new hotfix release:
> >> > > > > >> >> > > > > > > > >>
> >> > > > > >> >> > > > > > > > >>    - Heap-based timers’ restore behaviour
> >> is
> >> > > > > causing a
> >> > > > > >> >> > > critical
> >> > > > > >> >> > > > > > recovery
> >> > > > > >> >> > > > > > > > >>    issue for StateFun [1] [2] [3].
> >> > > > > >> >> > > > > > > > >>    - There are several robustness issues
> >> for
> >> > the
> >> > > > > >> FLIP-27
> >> > > > > >> >> new
> >> > > > > >> >> > > > > source
> >> > > > > >> >> > > > > > API,
> >> > > > > >> >> > > > > > > > >>    such as [4]. We already have some
> users
> >> > using
> >> > > > the
> >> > > > > >> >> FLIP-27
> >> > > > > >> >> > > API
> >> > > > > >> >> > > > > > with
> >> > > > > >> >> > > > > > > > >> 1.11.x,
> >> > > > > >> >> > > > > > > > >>    so it would be important to get those
> >> fixes
> >> > > in
> >> > > > > for
> >> > > > > >> >> 1.11.x
> >> > > > > >> >> > > as
> >> > > > > >> >> > > > > > well.
> >> > > > > >> >> > > > > > > > >>
> >> > > > > >> >> > > > > > > > >> Apart from the issues that are already
> >> marked
> >> > as
> >> > > > > >> blocker
> >> > > > > >> >> for
> >> > > > > >> >> > > > > 1.11.3
> >> > > > > >> >> > > > > > in
> >> > > > > >> >> > > > > > > > our
> >> > > > > >> >> > > > > > > > >> JIRA [5], please let us know in this
> >> thread if
> >> > > > there
> >> > > > > >> is
> >> > > > > >> >> > > already
> >> > > > > >> >> > > > > > ongoing
> >> > > > > >> >> > > > > > > > >> work for other important fixes that we
> >> should
> >> > > try
> >> > > > to
> >> > > > > >> >> > include.
> >> > > > > >> >> > > > > > > > >>
> >> > > > > >> >> > > > > > > > >> Xintong and I would like to volunteer for
> >> > > managing
> >> > > > > >> this
> >> > > > > >> >> > > release,
> >> > > > > >> >> > > > > and
> >> > > > > >> >> > > > > > > > will
> >> > > > > >> >> > > > > > > > >> try to communicate the priority of
> pending
> >> > > > blockers
> >> > > > > >> over
> >> > > > > >> >> the
> >> > > > > >> >> > > > next
> >> > > > > >> >> > > > > > few
> >> > > > > >> >> > > > > > > > >> days.
> >> > > > > >> >> > > > > > > > >> Since the aforementioned issues are quite
> >> > > > critical,
> >> > > > > >> we’d
> >> > > > > >> >> > like
> >> > > > > >> >> > > to
> >> > > > > >> >> > > > > aim
> >> > > > > >> >> > > > > > > > >> for a *feature
> >> > > > > >> >> > > > > > > > >> freeze by the end of next week (Oct.
> 30th)*
> >> > and
> >> > > > > start
> >> > > > > >> the
> >> > > > > >> >> > > > release
> >> > > > > >> >> > > > > > voting
> >> > > > > >> >> > > > > > > > >> process the week after.
> >> > > > > >> >> > > > > > > > >> If that is too short of a notice and you
> >> might
> >> > > > need
> >> > > > > >> more
> >> > > > > >> >> > time,
> >> > > > > >> >> > > > > > please
> >> > > > > >> >> > > > > > > > let
> >> > > > > >> >> > > > > > > > >> us know!
> >> > > > > >> >> > > > > > > > >>
> >> > > > > >> >> > > > > > > > >> Cheers,
> >> > > > > >> >> > > > > > > > >> Gordon
> >> > > > > >> >> > > > > > > > >>
> >> > > > > >> >> > > > > > > > >> [1]
> >> > > > > https://issues.apache.org/jira/browse/FLINK-19692
> >> > > > > >> >> > > > > > > > >> [2]
> >> > > > > https://issues.apache.org/jira/browse/FLINK-19741
> >> > > > > >> >> > > > > > > > >> [3]
> >> > > > > https://issues.apache.org/jira/browse/FLINK-19748
> >> > > > > >> >> > > > > > > > >> [4]
> >> > > > > https://issues.apache.org/jira/browse/FLINK-19717
> >> > > > > >> >> > > > > > > > >> [5]
> >> > > > > >> >> > > > > > > > >>
> >> > > > > >> >> > > > > > > > >>
> >> > > > > >> >> > > > > > > >
> >> > > > > >> >> > > > > >
> >> > > > > >> >> > > > >
> >> > > > > >> >> > > >
> >> > > > > >> >> > >
> >> > > > > >> >> >
> >> > > > > >> >>
> >> > > > > >>
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
> https://issues.apache.org/jira/issues/?jql=project%20%3D%20FLINK%20AND%20priority%20%3D%20Blocker%20AND%20fixVersion%20%3D%201.11.3
> >> > > > > >> >> > > > > > > > >>
> >> > > > > >> >> > > > > > > > >
> >> > > > > >> >> > > > > > > >
> >> > > > > >> >> > > > > >
> >> > > > > >> >> > > > >
> >> > > > > >> >> > > >
> >> > > > > >> >> > >
> >> > > > > >> >> >
> >> > > > > >> >> >
> >> > > > > >> >> > --
> >> > > > > >> >> > Best, Jingsong Lee
> >> > > > > >> >> >
> >> > > > > >> >>
> >> > > > > >> >
> >> > > > > >>
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
> >
>

Re: [DISCUSS] Releasing Apache Flink 1.11.3

Posted by "Tzu-Li (Gordon) Tai" <tz...@apache.org>.
Thanks for the efforts so far with pushing for 1.11.3.

I'd like to provide a mid-week update on how we're looking with all the
pending blockers and backports:

*Blockers**:*

Currently all blockers either have an open PR, or have been merged. I'll
highlight below the blockers *that still require reviewing efforts for them
to move forward:*

- [FLINK-19909] Flink application in attach mode could not terminate when
the only job is canceled. PR: https://github.com/apache/flink/pull/13911
- [FLINK-19717] SourceReaderBase.pollNext may return END_OF_INPUT if
SplitReader.fetch throws. PR: https://github.com/apache/flink/pull/13776

The above PRs currently have no reviews at all yet. They seem to already
have designated reviewers.


*Backports:*
- FLIP-27 robustness improvement backports: Stephan is currently working on
backporting several FLIP-27 changes. There are no PRs yet for the
backported changes.
- DataStreamUtils.collect() refactorings backport: Steven Wu mentioned to
backport this, but AFAIK this isn't assigned to anyone yet.

*ETAs*:

I'd like to request ETAs for the remaining backports, to prevent a creep in
the scope of this bugfix release.

We already have Flink users that would benefit from fixes that have been
merged to release-1.11 already, so technically speaking the backports
should be considered "nice-to-have" (to the best of my knowledge of the
changes) and could potentially be moved to a follow-up 1.11.4.
Most notably, the Stateful Functions project is already waiting on Flink
1.11.3 to address critical recovery issues (please see the StateFun 2.2.1
release discussion thread [1]).

@Stephan Ewen <se...@apache.org> @Becket Qin <be...@gmail.com> could
you provide an ETA for the FLIP-27 backports? It would help to get a better
estimate to decide how we proceed here.

Cheers,
Gordon

[1]
http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Releasing-StateFun-hotfix-version-2-2-1-td46239.html

On Wed, Nov 4, 2020 at 3:16 PM Tzu-Li (Gordon) Tai <tz...@apache.org>
wrote:

> > The collect() utils can be picked back, I see no issue with that (it is
> isolated utilities).
>
> Just checking on all the requested backports mentioned in this thread, and
> figuring out which ones seem to still be unassigned / open.
>
> Is someone working on backporting
> https://github.com/apache/flink/commit/09a7a66b7313fea64817fe960a8da1265b428efc
> to release-1.11 at the moment?
>
> On Wed, Nov 4, 2020 at 1:39 AM Steven Wu <st...@gmail.com> wrote:
>
>> @Stephan Ewen <se...@apache.org> yeah, we can do that. don't worry about
>> it. your earlier email had the perfect explanation on why file source
>> shouldn't be backported.
>>
>> On Tue, Nov 3, 2020 at 3:37 AM Stephan Ewen <se...@apache.org> wrote:
>>
>> > @Steven would it be possible to initially copy some of the code into the
>> > iceberg source and later replace it by a dependency on the Flink file
>> > source?
>> >
>> > On Mon, Nov 2, 2020 at 8:33 PM Steven Wu <st...@gmail.com> wrote:
>> >
>> > > Stephan, thanks a lot for explaining the file connector. that makes
>> > sense.
>> > >
>> > > I was asking because we were trying to reuse some of the
>> implementations
>> > in
>> > > the file source for Iceberg source. Flink Iceberg source lives in the
>> > > Iceberg repo, which is not possible to code against the master branch
>> of
>> > > the Flink code.
>> > >
>> > > On Mon, Nov 2, 2020 at 3:31 AM Stephan Ewen <se...@apache.org> wrote:
>> > >
>> > > > Hi Steven!
>> > > >
>> > > > So far there are no plans to pick back the file system connector
>> code.
>> > > This
>> > > > is still evolving and not finalized for 1.12, so I don't feel it is
>> a
>> > > good
>> > > > candidate to be backported.
>> > > > However, with the base connector changes backported, you should be
>> able
>> > > to
>> > > > run the file connector code from master against 1.11.3.
>> > > >
>> > > > The collect() utils can be picked back, I see no issue with that
>> (it is
>> > > > isolated utilities).
>> > > >
>> > > > Best,
>> > > > Stephan
>> > > >
>> > > >
>> > > > On Mon, Nov 2, 2020 at 3:02 AM Steven Wu <st...@gmail.com>
>> wrote:
>> > > >
>> > > > > Basically, it would be great to get the latest code in the
>> > > > > flink-connector-files (FLIP-27).
>> > > > >
>> > > > > On Sat, Oct 31, 2020 at 9:57 AM Steven Wu <st...@gmail.com>
>> > > wrote:
>> > > > >
>> > > > > > Stephan, it will be great if we can also backport the
>> > DataStreamUtils
>> > > > > > related commits that help with collecting output from unbounded
>> > > > streams.
>> > > > > > e.g.
>> > > > > >
>> > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> https://github.com/apache/flink/commit/09a7a66b7313fea64817fe960a8da1265b428efc
>> > > > > >
>> > > > > > I tried to copy and paste the code to unblock myself. but it
>> > quickly
>> > > > got
>> > > > > > into the rabbit hole of more and more code.
>> > > > > >
>> > > > > > On Fri, Oct 30, 2020 at 11:02 AM Stephan Ewen <sewen@apache.org
>> >
>> > > > wrote:
>> > > > > >
>> > > > > >> I have started with backporting the source API changes. Some
>> minor
>> > > > > >> conflicts to solve, will need a bit more to finish this.
>> > > > > >>
>> > > > > >> On Fri, Oct 30, 2020 at 7:25 AM Tzu-Li (Gordon) Tai <
>> > > > > tzulitai@apache.org>
>> > > > > >> wrote:
>> > > > > >>
>> > > > > >> > @Stephan Ewen <se...@apache.org>
>> > > > > >> > Are there already plans or ongoing efforts for backporting
>> the
>> > > list
>> > > > of
>> > > > > >> > FLIP-27 changes that you posted?
>> > > > > >> >
>> > > > > >> > On Thu, Oct 29, 2020 at 7:08 PM Xintong Song <
>> > > tonysong820@gmail.com
>> > > > >
>> > > > > >> > wrote:
>> > > > > >> >
>> > > > > >> >> Hi folks,
>> > > > > >> >>
>> > > > > >> >> Just to provide some updates concerning the status on the
>> > > > > >> >> test instabilities.
>> > > > > >> >>
>> > > > > >> >> Currently, we have 30 unresolved tickets labeled with
>> `Affects
>> > > > > Version`
>> > > > > >> >> 1.11.x.
>> > > > > >> >>
>> > > > > >> >>
>> > > > > >>
>> > > > >
>> > > >
>> > >
>> >
>> https://issues.apache.org/jira/browse/FLINK-19775?filter=12348580&jql=project%20%3D%20FLINK%20AND%20resolution%20%3D%20Unresolved%20AND%20affectedVersion%20in%20(1.11.0%2C%201.11.1%2C%201.11.2%2C%201.11.3)%20AND%20labels%20%3D%20test-stability%20ORDER%20BY%20created%20DESC
>> > > > > >> >>
>> > > > > >> >> Among the 30 tickets, 11 of them are:
>> > > > > >> >> - Have occured in the recent 3 months
>> > > > > >> >> - Not confirmed to be pure testability issues
>> > > > > >> >> - Not confirmed to be rare condition cases
>> > > > > >> >>
>> > > > > >> >> It would be nice if someone familiar with these components
>> can
>> > > > take a
>> > > > > >> look
>> > > > > >> >> into these issues.
>> > > > > >> >>
>> > > > > >> >> - https://issues.apache.org/jira/browse/FLINK-17159 (ES6)
>> > > > > >> >> - https://issues.apache.org/jira/browse/FLINK-17912 (Kafka)
>> > > > > >> >> - https://issues.apache.org/jira/browse/FLINK-17949 (Kafka)
>> > > > > >> >> ⁃ https://issues.apache.org/jira/browse/FLINK-18444 (Kafka)
>> > > > > >> >> - https://issues.apache.org/jira/browse/FLINK-18634 (Kafka)
>> > > > > >> >> - https://issues.apache.org/jira/browse/FLINK-18648 (Kafka)
>> > > > > >> >> - https://issues.apache.org/jira/browse/FLINK-18807 (Kafka)
>> > > > > >> >> - https://issues.apache.org/jira/browse/FLINK-19369
>> > > > (BlobClientTest)
>> > > > > >> >> - https://issues.apache.org/jira/browse/FLINK-19436 (TPCDS)
>> > > > > >> >> - https://issues.apache.org/jira/browse/FLINK-19690
>> > > > (Format/Parquet)
>> > > > > >> >> - https://issues.apache.org/jira/browse/FLINK-19775
>> > > > > >> >> (SystemProcessingTimeServiceTest)
>> > > > > >> >>
>> > > > > >> >> Thank you~
>> > > > > >> >>
>> > > > > >> >> Xintong Song
>> > > > > >> >>
>> > > > > >> >>
>> > > > > >> >>
>> > > > > >> >> On Thu, Oct 29, 2020 at 10:21 AM Jingsong Li <
>> > > > jingsonglee0@gmail.com
>> > > > > >
>> > > > > >> >> wrote:
>> > > > > >> >>
>> > > > > >> >> > +1 to backport the FLIP-27 adjustments to 1.11.x.
>> > > > > >> >> >
>> > > > > >> >> > If possible, that would be great. Many people are looking
>> > > forward
>> > > > > to
>> > > > > >> the
>> > > > > >> >> > FLIP-27 interface, but they don't want to take the risk to
>> > > > upgrade
>> > > > > to
>> > > > > >> >> 1.12
>> > > > > >> >> > (And wait 1.12). After all, 1.11 is a relatively stable
>> > > version.
>> > > > > >> >> >
>> > > > > >> >> > Best,
>> > > > > >> >> > Jingsong
>> > > > > >> >> >
>> > > > > >> >> > On Thu, Oct 29, 2020 at 1:24 AM Stephan Ewen <
>> > sewen@apache.org
>> > > >
>> > > > > >> wrote:
>> > > > > >> >> >
>> > > > > >> >> > > Thanks for starting this.
>> > > > > >> >> > >
>> > > > > >> >> > > +1 form my side to backport the FLIP-27 adjustments to
>> > > 1.11.x.
>> > > > > >> >> > >
>> > > > > >> >> > > There were quite a few changes, and I think we need to
>> > > > > cherry-pick
>> > > > > >> >> them
>> > > > > >> >> > all
>> > > > > >> >> > > to not get some inconsistent mix of changes and many
>> merge
>> > > > > >> conflicts.
>> > > > > >> >> > > I made a list below of what we need to add to
>> > "release-1.11".
>> > > > > >> >> > >
>> > > > > >> >> > > *#### Core Source API Changes to backport (in REVERSE
>> > order)*
>> > > > > >> >> > >
>> > > > > >> >> > >   (Use: "git log
>> > > > > >> >> > >
>> > > > flink-core/src/main/java/org/apache/flink/api/connector/source")
>> > > > > >> >> > >
>> > > > > >> >> > > commit 162c072e9265a7b6dd9d6f5459eb7974091c4c4e
>> > > > > >> >> > > [FLINK-19492][core] Consolidate Source Events between
>> > Source
>> > > > API
>> > > > > >> and
>> > > > > >> >> > Split
>> > > > > >> >> > > Reader API
>> > > > > >> >> > >
>> > > > > >> >> > > commit 428fbbad85379f71a9fbf1a1ab3e3dcfa9006436
>> > > > > >> >> > > [FLINK-19205][core] Add access to configuration and
>> > hostname
>> > > in
>> > > > > the
>> > > > > >> >> > > SourceReaderContext
>> > > > > >> >> > >
>> > > > > >> >> > > commit 723e1790744ddc78d19c7c978442af1383f38d33
>> > > > > >> >> > > [hotfix][core] Add to Source Enumerator convenience
>> methods
>> > > to
>> > > > > >> assign
>> > > > > >> >> > > single split
>> > > > > >> >> > >
>> > > > > >> >> > > *#### Connector Base Changes to Backport (in REVERSE
>> > order)*
>> > > > > >> >> > >
>> > > > > >> >> > >   (Use: "git log flink-connectors/flink-connector-base")
>> > > > > >> >> > >
>> > > > > >> >> > > commit 401f56fe9d6b0271260edf9787cdcbfe4d03874d
>> > > > > >> >> > > [FLINK-19427][FLINK-19489][tests] Fix test conditions
>> for
>> > > > > >> >> > > 'SplitFetcherTest.testNotifiesWhenGoingIdleConcurrent()'
>> > > > > >> >> > >
>> > > > > >> >> > > commit 68c5c2ff779d82a1ff81ffaf60d8a1b283797db1
>> > > > > >> >> > > [FLINK-19448][connector base] Explicitly check for
>> > > un-expected
>> > > > > >> >> condition
>> > > > > >> >> > > that would leave an inconsistent state
>> > > > > >> >> > >
>> > > > > >> >> > > commit 162c072e9265a7b6dd9d6f5459eb7974091c4c4e
>> > > > > >> >> > > [FLINK-19492][core] Consolidate Source Events between
>> > Source
>> > > > API
>> > > > > >> and
>> > > > > >> >> > Split
>> > > > > >> >> > > Reader API
>> > > > > >> >> > >
>> > > > > >> >> > > commit c1ca7a4c7c21ec8868c14cdddd43c559625b794c
>> > > > > >> >> > > [refactor][tests] Move some source test utils from
>> > > > > >> >> flink-connector-base
>> > > > > >> >> > to
>> > > > > >> >> > > flink-core
>> > > > > >> >> > >
>> > > > > >> >> > > commit ee5c4c211c35c70d28252363bbc8400453609977
>> > > > > >> >> > > [FLINK-19251][connectors] Avoid confusing queue
>> handling in
>> > > > > >> >> > > "SplitReader.handleSplitsChanges()"
>> > > > > >> >> > >
>> > > > > >> >> > > commit 5abef56b2bf85bcac786f6b16b6899b6cced7176
>> > > > > >> >> > > [FLINK-19250][connectors] Fix error propagation in
>> > connector
>> > > > base
>> > > > > >> >> > > (SplitFetcherManager).
>> > > > > >> >> > >
>> > > > > >> >> > > commit 8fcca837c55a9216595ee4c03038b52747098dbb
>> > > > > >> >> > > [hotfix][connectors] Improve JavaDocs for
>> > > > > >> SingleThreadFetcherManager
>> > > > > >> >> > >
>> > > > > >> >> > > commit 4700bb5dde3303cbe98882f6beb7379425717b01
>> > > > > >> >> > > [FLINK-19225][connectors] Various small improvements to
>> > > > > >> >> SourceReaderBase
>> > > > > >> >> > > (part 2)
>> > > > > >> >> > >
>> > > > > >> >> > > commit 12261c6b7ed6478a9b9f6a69cb58246b83cab9b7
>> > > > > >> >> > > [FLINK-17393][connectors] (follow-up) Wakeup the
>> > > SplitFetchers
>> > > > > more
>> > > > > >> >> > > elegantly.
>> > > > > >> >> > >
>> > > > > >> >> > > commit c60aaff0249bfd6b5871b7f82e03efc487a54d6b
>> > > > > >> >> > > [hotfix][tests] Extend test coverage for
>> > > > > >> FutureCompletingBlockingQueue
>> > > > > >> >> > >
>> > > > > >> >> > > commit cef8a587d7fd2fe64cc644da5ed095d82e46f631
>> > > > > >> >> > > [FLINK-19245][connectors] Set default capacity for
>> > > > > >> >> > > FutureCompletingBlockingQueue.
>> > > > > >> >> > >
>> > > > > >> >> > > commit 4ea95782b4c6a2538153d4d16ad3f4839c7de0fb
>> > > > > >> >> > > [FLINK-19223][connectors] Simplify Availability Future
>> > Model
>> > > in
>> > > > > >> Base
>> > > > > >> >> > > Connector
>> > > > > >> >> > >
>> > > > > >> >> > > commit 511857049ba30c8ff0ee56da551fa4a479dc583e
>> > > > > >> >> > > [FLINK-18128][connectors] Ensure idle split fetchers
>> lead
>> > to
>> > > > > >> >> availability
>> > > > > >> >> > > notifications.
>> > > > > >> >> > >
>> > > > > >> >> > > commit a8206467af0830dcb89623ea068b5ca3b3450c92
>> > > > > >> >> > > [refactor][core] Eagerly initialize the FetchTask to
>> > support
>> > > > > proper
>> > > > > >> >> unit
>> > > > > >> >> > > testing
>> > > > > >> >> > >
>> > > > > >> >> > > commit 3b2f54bcb437f98e6137c904045cc51072b5c06b
>> > > > > >> >> > > [hotfix][tests] Move constants in SplitFetcherTest
>> relevant
>> > > to
>> > > > > only
>> > > > > >> >> one
>> > > > > >> >> > > test into test method
>> > > > > >> >> > >
>> > > > > >> >> > > commit d7625760a75a508bf05bcddc380bb4d62ee1743e
>> > > > > >> >> > > [FLINK-19225][connectors] Various small improvements to
>> > > > > >> >> SourceReaderBase
>> > > > > >> >> > >
>> > > > > >> >> > > commit a5b0d3297748c1be47ad579a88f24df2255a8df1
>> > > > > >> >> > > [FLINK-17393][connectors] Wakeup the SplitFetchers more
>> > > > > elegantly.
>> > > > > >> >> > >
>> > > > > >> >> > > commit f42a3ebc3e81a034b7221a803c153636fef34903
>> > > > > >> >> > > [FLINK-18680][connectors] Make connector base
>> > > > RecordsWithSplitIds
>> > > > > >> more
>> > > > > >> >> > > lightweight.
>> > > > > >> >> > >
>> > > > > >> >> > > commit e3d273de822b085183d09b275a445879ff94b350
>> > > > > >> >> > > [FLINK-19162][connectors] Add 'recycle()' to the
>> > > > > >> RecordsWithSplitIds
>> > > > > >> >> to
>> > > > > >> >> > > support reuse of heavy objects.
>> > > > > >> >> > >
>> > > > > >> >> > > commit 8ebc464c2520453a70001cd712abc8dee6ee89e0
>> > > > > >> >> > > [hotfix][testing] Add a set of parameterizable testing
>> > mocks
>> > > > for
>> > > > > >> the
>> > > > > >> >> > Split
>> > > > > >> >> > > Reader API
>> > > > > >> >> > >
>> > > > > >> >> > > commit 930a07438be1185388d7150640f294dfe2a5d378
>> > > > > >> >> > > [hotfix][connectors] Add RequestSplitEvent to
>> > > > > >> 'flink-connector-base'
>> > > > > >> >> > >
>> > > > > >> >> > > commit e945ce8a933bc378844782f784ca473c767ca159
>> > > > > >> >> > > [refactor][connectors] Backport of the connector-base
>> > > exception
>> > > > > >> >> handling
>> > > > > >> >> > > from the Kafka Connector Pull Request
>> > > > > >> >> > >
>> > > > > >> >> > >
>> > > > > >> >> > > On Wed, Oct 28, 2020 at 5:49 PM Steven Wu <
>> > > > stevenz3wu@gmail.com>
>> > > > > >> >> wrote:
>> > > > > >> >> > >
>> > > > > >> >> > > > I would love to see this FLIP-27 source interface
>> > > improvement
>> > > > > [1]
>> > > > > >> >> made
>> > > > > >> >> > to
>> > > > > >> >> > > > 1.11.3.
>> > > > > >> >> > > >
>> > > > > >> >> > > > [1] https://issues.apache.org/jira/browse/FLINK-19698
>> > > > > >> >> > > >
>> > > > > >> >> > > > On Wed, Oct 28, 2020 at 12:32 AM Tzu-Li (Gordon) Tai <
>> > > > > >> >> > > tzulitai@apache.org>
>> > > > > >> >> > > > wrote:
>> > > > > >> >> > > >
>> > > > > >> >> > > > > Thanks for the replies so far!
>> > > > > >> >> > > > >
>> > > > > >> >> > > > > Just to provide a brief update on the status of
>> > blockers
>> > > > for
>> > > > > >> >> 1.11.3
>> > > > > >> >> > so
>> > > > > >> >> > > > far:
>> > > > > >> >> > > > >
>> > > > > >> >> > > > >
>> > > > > >> >> > > > > *PR opened, pending reviewer*- [FLINK-19717]
>> > > > > >> >> > SourceReaderBase.pollNext
>> > > > > >> >> > > > may
>> > > > > >> >> > > > > return END_OF_INPUT if SplitReader.fetch throws (
>> > > > > >> >> > > > > https://github.com/apache/flink/pull/13776)
>> > > > > >> >> > > > >
>> > > > > >> >> > > > > *PR opened, reviewed + close to being merged*
>> > > > > >> >> > > > > - [FLINK-19741] Timer service should skip restoring
>> > from
>> > > > raw
>> > > > > >> keyed
>> > > > > >> >> > > stream
>> > > > > >> >> > > > > if it isn't the writer (
>> > > > > >> >> https://github.com/apache/flink/pull/13761)
>> > > > > >> >> > > > > - [FLINK-19748] Raw keyed stream key group iterator
>> > > should
>> > > > be
>> > > > > >> >> > skipping
>> > > > > >> >> > > > > unwritten key groups (
>> > > > > >> https://github.com/apache/flink/pull/13772)
>> > > > > >> >> > > > >
>> > > > > >> >> > > > > *Merged*
>> > > > > >> >> > > > > - [FLINK-19154] Application mode deletes HA data in
>> > case
>> > > of
>> > > > > >> >> suspended
>> > > > > >> >> > > > > ZooKeeper connection
>> > > > > >> >> > > > > - [FLINK-19569] Upgrade ICU4J to 67.1+
>> > > > > >> >> > > > >
>> > > > > >> >> > > > > Right now as it seems, progress is mainly blocked
>> on a
>> > > > > reviewer
>> > > > > >> >> for
>> > > > > >> >> > > > > FLINK-19717.
>> > > > > >> >> > > > > Meanwhile, Xintong is keeping an eye on test
>> > > instabilities
>> > > > > [1]
>> > > > > >> to
>> > > > > >> >> see
>> > > > > >> >> > > if
>> > > > > >> >> > > > > there are any fixes that should be applied to
>> > > > `release-1.11`.
>> > > > > >> >> > > > >
>> > > > > >> >> > > > > This is also a reminder, that if there are other
>> > blockers
>> > > > > that
>> > > > > >> we
>> > > > > >> >> > need
>> > > > > >> >> > > to
>> > > > > >> >> > > > > be aware of, or a need to re-establish estimated
>> time
>> > for
>> > > > > >> getting
>> > > > > >> >> > fixes
>> > > > > >> >> > > > in
>> > > > > >> >> > > > > and delay the RC for 1.11.3, please do let us know!
>> > > > > >> >> > > > >
>> > > > > >> >> > > > > Cheers,
>> > > > > >> >> > > > > Gordon
>> > > > > >> >> > > > >
>> > > > > >> >> > > > > [1]
>> > > > > >> >> > > > >
>> > > > > >> >> > > > >
>> > > > > >> >> > > >
>> > > > > >> >> > >
>> > > > > >> >> >
>> > > > > >> >>
>> > > > > >>
>> > > > >
>> > > >
>> > >
>> >
>> https://issues.apache.org/jira/issues/?jql=project%20%3D%20FLINK%20AND%20status%20in%20(Open%2C%20%22In%20Progress%22%2C%20Reopened)%20AND%20affectedVersion%20in%20(1.11.0%2C%201.11.1%2C%201.11.2)%20AND%20labels%20%3D%20test-stability
>> > > > > >> >> > > > >
>> > > > > >> >> > > > > On Mon, Oct 26, 2020 at 9:43 PM Kostas Kloudas <
>> > > > > >> >> kkloudas@apache.org>
>> > > > > >> >> > > > > wrote:
>> > > > > >> >> > > > >
>> > > > > >> >> > > > > > +1 for releasing Flink 1.11.3 as it contains a
>> number
>> > > of
>> > > > > >> >> important
>> > > > > >> >> > > > > > fixes and thanks Gordon and Xintong for
>> volunteering.
>> > > > > >> >> > > > > >
>> > > > > >> >> > > > > > Cheers,
>> > > > > >> >> > > > > > Kostas
>> > > > > >> >> > > > > >
>> > > > > >> >> > > > > > On Mon, Oct 26, 2020 at 4:37 AM Yu Li <
>> > > carp84@gmail.com>
>> > > > > >> wrote:
>> > > > > >> >> > > > > > >
>> > > > > >> >> > > > > > > +1 for releasing Flink 1.11.3, and thanks Gordon
>> > and
>> > > > > >> Xintong
>> > > > > >> >> for
>> > > > > >> >> > > > > > > volunteering as our release managers.
>> > > > > >> >> > > > > > >
>> > > > > >> >> > > > > > > Best Regards,
>> > > > > >> >> > > > > > > Yu
>> > > > > >> >> > > > > > >
>> > > > > >> >> > > > > > >
>> > > > > >> >> > > > > > > On Mon, 26 Oct 2020 at 09:45, Xintong Song <
>> > > > > >> >> > tonysong820@gmail.com>
>> > > > > >> >> > > > > > wrote:
>> > > > > >> >> > > > > > >
>> > > > > >> >> > > > > > > > Thanks Gordan for starting this discussion.
>> > > > > >> >> > > > > > > > My pleasure to help with the release process.
>> > > > > >> >> > > > > > > >
>> > > > > >> >> > > > > > > > Thank you~
>> > > > > >> >> > > > > > > >
>> > > > > >> >> > > > > > > > Xintong Song
>> > > > > >> >> > > > > > > >
>> > > > > >> >> > > > > > > >
>> > > > > >> >> > > > > > > >
>> > > > > >> >> > > > > > > > On Fri, Oct 23, 2020 at 11:29 PM Till
>> Rohrmann <
>> > > > > >> >> > > > trohrmann@apache.org
>> > > > > >> >> > > > > >
>> > > > > >> >> > > > > > > > wrote:
>> > > > > >> >> > > > > > > >
>> > > > > >> >> > > > > > > > > Thanks for starting this discussion Gordon.
>> > There
>> > > > are
>> > > > > >> over
>> > > > > >> >> > 100
>> > > > > >> >> > > > > issues
>> > > > > >> >> > > > > > > > > which are fixed for 1.11.3. Hence +1 for a
>> > > soonish
>> > > > > >> 1.11.3
>> > > > > >> >> > > > release.
>> > > > > >> >> > > > > > Thanks
>> > > > > >> >> > > > > > > > > for volunteering as our release managers
>> Gordon
>> > > and
>> > > > > >> >> Xintong!
>> > > > > >> >> > > > > > > > >
>> > > > > >> >> > > > > > > > > Cheers,
>> > > > > >> >> > > > > > > > > Till
>> > > > > >> >> > > > > > > > >
>> > > > > >> >> > > > > > > > > On Fri, Oct 23, 2020 at 5:02 PM Tzu-Li
>> (Gordon)
>> > > > Tai <
>> > > > > >> >> > > > > > tzulitai@apache.org
>> > > > > >> >> > > > > > > > >
>> > > > > >> >> > > > > > > > > wrote:
>> > > > > >> >> > > > > > > > >
>> > > > > >> >> > > > > > > > >> Hi,
>> > > > > >> >> > > > > > > > >>
>> > > > > >> >> > > > > > > > >> Xintong and I would like to start a
>> discussion
>> > > for
>> > > > > >> >> releasing
>> > > > > >> >> > > > Flink
>> > > > > >> >> > > > > > > > 1.11.3
>> > > > > >> >> > > > > > > > >> soon.
>> > > > > >> >> > > > > > > > >>
>> > > > > >> >> > > > > > > > >> It seems like we already have a few
>> pressing
>> > > > issues
>> > > > > >> that
>> > > > > >> >> > needs
>> > > > > >> >> > > > to
>> > > > > >> >> > > > > be
>> > > > > >> >> > > > > > > > >> included in a new hotfix release:
>> > > > > >> >> > > > > > > > >>
>> > > > > >> >> > > > > > > > >>    - Heap-based timers’ restore behaviour
>> is
>> > > > > causing a
>> > > > > >> >> > > critical
>> > > > > >> >> > > > > > recovery
>> > > > > >> >> > > > > > > > >>    issue for StateFun [1] [2] [3].
>> > > > > >> >> > > > > > > > >>    - There are several robustness issues
>> for
>> > the
>> > > > > >> FLIP-27
>> > > > > >> >> new
>> > > > > >> >> > > > > source
>> > > > > >> >> > > > > > API,
>> > > > > >> >> > > > > > > > >>    such as [4]. We already have some users
>> > using
>> > > > the
>> > > > > >> >> FLIP-27
>> > > > > >> >> > > API
>> > > > > >> >> > > > > > with
>> > > > > >> >> > > > > > > > >> 1.11.x,
>> > > > > >> >> > > > > > > > >>    so it would be important to get those
>> fixes
>> > > in
>> > > > > for
>> > > > > >> >> 1.11.x
>> > > > > >> >> > > as
>> > > > > >> >> > > > > > well.
>> > > > > >> >> > > > > > > > >>
>> > > > > >> >> > > > > > > > >> Apart from the issues that are already
>> marked
>> > as
>> > > > > >> blocker
>> > > > > >> >> for
>> > > > > >> >> > > > > 1.11.3
>> > > > > >> >> > > > > > in
>> > > > > >> >> > > > > > > > our
>> > > > > >> >> > > > > > > > >> JIRA [5], please let us know in this
>> thread if
>> > > > there
>> > > > > >> is
>> > > > > >> >> > > already
>> > > > > >> >> > > > > > ongoing
>> > > > > >> >> > > > > > > > >> work for other important fixes that we
>> should
>> > > try
>> > > > to
>> > > > > >> >> > include.
>> > > > > >> >> > > > > > > > >>
>> > > > > >> >> > > > > > > > >> Xintong and I would like to volunteer for
>> > > managing
>> > > > > >> this
>> > > > > >> >> > > release,
>> > > > > >> >> > > > > and
>> > > > > >> >> > > > > > > > will
>> > > > > >> >> > > > > > > > >> try to communicate the priority of pending
>> > > > blockers
>> > > > > >> over
>> > > > > >> >> the
>> > > > > >> >> > > > next
>> > > > > >> >> > > > > > few
>> > > > > >> >> > > > > > > > >> days.
>> > > > > >> >> > > > > > > > >> Since the aforementioned issues are quite
>> > > > critical,
>> > > > > >> we’d
>> > > > > >> >> > like
>> > > > > >> >> > > to
>> > > > > >> >> > > > > aim
>> > > > > >> >> > > > > > > > >> for a *feature
>> > > > > >> >> > > > > > > > >> freeze by the end of next week (Oct. 30th)*
>> > and
>> > > > > start
>> > > > > >> the
>> > > > > >> >> > > > release
>> > > > > >> >> > > > > > voting
>> > > > > >> >> > > > > > > > >> process the week after.
>> > > > > >> >> > > > > > > > >> If that is too short of a notice and you
>> might
>> > > > need
>> > > > > >> more
>> > > > > >> >> > time,
>> > > > > >> >> > > > > > please
>> > > > > >> >> > > > > > > > let
>> > > > > >> >> > > > > > > > >> us know!
>> > > > > >> >> > > > > > > > >>
>> > > > > >> >> > > > > > > > >> Cheers,
>> > > > > >> >> > > > > > > > >> Gordon
>> > > > > >> >> > > > > > > > >>
>> > > > > >> >> > > > > > > > >> [1]
>> > > > > https://issues.apache.org/jira/browse/FLINK-19692
>> > > > > >> >> > > > > > > > >> [2]
>> > > > > https://issues.apache.org/jira/browse/FLINK-19741
>> > > > > >> >> > > > > > > > >> [3]
>> > > > > https://issues.apache.org/jira/browse/FLINK-19748
>> > > > > >> >> > > > > > > > >> [4]
>> > > > > https://issues.apache.org/jira/browse/FLINK-19717
>> > > > > >> >> > > > > > > > >> [5]
>> > > > > >> >> > > > > > > > >>
>> > > > > >> >> > > > > > > > >>
>> > > > > >> >> > > > > > > >
>> > > > > >> >> > > > > >
>> > > > > >> >> > > > >
>> > > > > >> >> > > >
>> > > > > >> >> > >
>> > > > > >> >> >
>> > > > > >> >>
>> > > > > >>
>> > > > >
>> > > >
>> > >
>> >
>> https://issues.apache.org/jira/issues/?jql=project%20%3D%20FLINK%20AND%20priority%20%3D%20Blocker%20AND%20fixVersion%20%3D%201.11.3
>> > > > > >> >> > > > > > > > >>
>> > > > > >> >> > > > > > > > >
>> > > > > >> >> > > > > > > >
>> > > > > >> >> > > > > >
>> > > > > >> >> > > > >
>> > > > > >> >> > > >
>> > > > > >> >> > >
>> > > > > >> >> >
>> > > > > >> >> >
>> > > > > >> >> > --
>> > > > > >> >> > Best, Jingsong Lee
>> > > > > >> >> >
>> > > > > >> >>
>> > > > > >> >
>> > > > > >>
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>>
>

Re: [DISCUSS] Releasing Apache Flink 1.11.3

Posted by "Tzu-Li (Gordon) Tai" <tz...@apache.org>.
> The collect() utils can be picked back, I see no issue with that (it is
isolated utilities).

Just checking on all the requested backports mentioned in this thread, and
figuring out which ones seem to still be unassigned / open.

Is someone working on backporting
https://github.com/apache/flink/commit/09a7a66b7313fea64817fe960a8da1265b428efc
to release-1.11 at the moment?

On Wed, Nov 4, 2020 at 1:39 AM Steven Wu <st...@gmail.com> wrote:

> @Stephan Ewen <se...@apache.org> yeah, we can do that. don't worry about
> it. your earlier email had the perfect explanation on why file source
> shouldn't be backported.
>
> On Tue, Nov 3, 2020 at 3:37 AM Stephan Ewen <se...@apache.org> wrote:
>
> > @Steven would it be possible to initially copy some of the code into the
> > iceberg source and later replace it by a dependency on the Flink file
> > source?
> >
> > On Mon, Nov 2, 2020 at 8:33 PM Steven Wu <st...@gmail.com> wrote:
> >
> > > Stephan, thanks a lot for explaining the file connector. that makes
> > sense.
> > >
> > > I was asking because we were trying to reuse some of the
> implementations
> > in
> > > the file source for Iceberg source. Flink Iceberg source lives in the
> > > Iceberg repo, which is not possible to code against the master branch
> of
> > > the Flink code.
> > >
> > > On Mon, Nov 2, 2020 at 3:31 AM Stephan Ewen <se...@apache.org> wrote:
> > >
> > > > Hi Steven!
> > > >
> > > > So far there are no plans to pick back the file system connector
> code.
> > > This
> > > > is still evolving and not finalized for 1.12, so I don't feel it is a
> > > good
> > > > candidate to be backported.
> > > > However, with the base connector changes backported, you should be
> able
> > > to
> > > > run the file connector code from master against 1.11.3.
> > > >
> > > > The collect() utils can be picked back, I see no issue with that (it
> is
> > > > isolated utilities).
> > > >
> > > > Best,
> > > > Stephan
> > > >
> > > >
> > > > On Mon, Nov 2, 2020 at 3:02 AM Steven Wu <st...@gmail.com>
> wrote:
> > > >
> > > > > Basically, it would be great to get the latest code in the
> > > > > flink-connector-files (FLIP-27).
> > > > >
> > > > > On Sat, Oct 31, 2020 at 9:57 AM Steven Wu <st...@gmail.com>
> > > wrote:
> > > > >
> > > > > > Stephan, it will be great if we can also backport the
> > DataStreamUtils
> > > > > > related commits that help with collecting output from unbounded
> > > > streams.
> > > > > > e.g.
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/flink/commit/09a7a66b7313fea64817fe960a8da1265b428efc
> > > > > >
> > > > > > I tried to copy and paste the code to unblock myself. but it
> > quickly
> > > > got
> > > > > > into the rabbit hole of more and more code.
> > > > > >
> > > > > > On Fri, Oct 30, 2020 at 11:02 AM Stephan Ewen <se...@apache.org>
> > > > wrote:
> > > > > >
> > > > > >> I have started with backporting the source API changes. Some
> minor
> > > > > >> conflicts to solve, will need a bit more to finish this.
> > > > > >>
> > > > > >> On Fri, Oct 30, 2020 at 7:25 AM Tzu-Li (Gordon) Tai <
> > > > > tzulitai@apache.org>
> > > > > >> wrote:
> > > > > >>
> > > > > >> > @Stephan Ewen <se...@apache.org>
> > > > > >> > Are there already plans or ongoing efforts for backporting the
> > > list
> > > > of
> > > > > >> > FLIP-27 changes that you posted?
> > > > > >> >
> > > > > >> > On Thu, Oct 29, 2020 at 7:08 PM Xintong Song <
> > > tonysong820@gmail.com
> > > > >
> > > > > >> > wrote:
> > > > > >> >
> > > > > >> >> Hi folks,
> > > > > >> >>
> > > > > >> >> Just to provide some updates concerning the status on the
> > > > > >> >> test instabilities.
> > > > > >> >>
> > > > > >> >> Currently, we have 30 unresolved tickets labeled with
> `Affects
> > > > > Version`
> > > > > >> >> 1.11.x.
> > > > > >> >>
> > > > > >> >>
> > > > > >>
> > > > >
> > > >
> > >
> >
> https://issues.apache.org/jira/browse/FLINK-19775?filter=12348580&jql=project%20%3D%20FLINK%20AND%20resolution%20%3D%20Unresolved%20AND%20affectedVersion%20in%20(1.11.0%2C%201.11.1%2C%201.11.2%2C%201.11.3)%20AND%20labels%20%3D%20test-stability%20ORDER%20BY%20created%20DESC
> > > > > >> >>
> > > > > >> >> Among the 30 tickets, 11 of them are:
> > > > > >> >> - Have occured in the recent 3 months
> > > > > >> >> - Not confirmed to be pure testability issues
> > > > > >> >> - Not confirmed to be rare condition cases
> > > > > >> >>
> > > > > >> >> It would be nice if someone familiar with these components
> can
> > > > take a
> > > > > >> look
> > > > > >> >> into these issues.
> > > > > >> >>
> > > > > >> >> - https://issues.apache.org/jira/browse/FLINK-17159 (ES6)
> > > > > >> >> - https://issues.apache.org/jira/browse/FLINK-17912 (Kafka)
> > > > > >> >> - https://issues.apache.org/jira/browse/FLINK-17949 (Kafka)
> > > > > >> >> ⁃ https://issues.apache.org/jira/browse/FLINK-18444 (Kafka)
> > > > > >> >> - https://issues.apache.org/jira/browse/FLINK-18634 (Kafka)
> > > > > >> >> - https://issues.apache.org/jira/browse/FLINK-18648 (Kafka)
> > > > > >> >> - https://issues.apache.org/jira/browse/FLINK-18807 (Kafka)
> > > > > >> >> - https://issues.apache.org/jira/browse/FLINK-19369
> > > > (BlobClientTest)
> > > > > >> >> - https://issues.apache.org/jira/browse/FLINK-19436 (TPCDS)
> > > > > >> >> - https://issues.apache.org/jira/browse/FLINK-19690
> > > > (Format/Parquet)
> > > > > >> >> - https://issues.apache.org/jira/browse/FLINK-19775
> > > > > >> >> (SystemProcessingTimeServiceTest)
> > > > > >> >>
> > > > > >> >> Thank you~
> > > > > >> >>
> > > > > >> >> Xintong Song
> > > > > >> >>
> > > > > >> >>
> > > > > >> >>
> > > > > >> >> On Thu, Oct 29, 2020 at 10:21 AM Jingsong Li <
> > > > jingsonglee0@gmail.com
> > > > > >
> > > > > >> >> wrote:
> > > > > >> >>
> > > > > >> >> > +1 to backport the FLIP-27 adjustments to 1.11.x.
> > > > > >> >> >
> > > > > >> >> > If possible, that would be great. Many people are looking
> > > forward
> > > > > to
> > > > > >> the
> > > > > >> >> > FLIP-27 interface, but they don't want to take the risk to
> > > > upgrade
> > > > > to
> > > > > >> >> 1.12
> > > > > >> >> > (And wait 1.12). After all, 1.11 is a relatively stable
> > > version.
> > > > > >> >> >
> > > > > >> >> > Best,
> > > > > >> >> > Jingsong
> > > > > >> >> >
> > > > > >> >> > On Thu, Oct 29, 2020 at 1:24 AM Stephan Ewen <
> > sewen@apache.org
> > > >
> > > > > >> wrote:
> > > > > >> >> >
> > > > > >> >> > > Thanks for starting this.
> > > > > >> >> > >
> > > > > >> >> > > +1 form my side to backport the FLIP-27 adjustments to
> > > 1.11.x.
> > > > > >> >> > >
> > > > > >> >> > > There were quite a few changes, and I think we need to
> > > > > cherry-pick
> > > > > >> >> them
> > > > > >> >> > all
> > > > > >> >> > > to not get some inconsistent mix of changes and many
> merge
> > > > > >> conflicts.
> > > > > >> >> > > I made a list below of what we need to add to
> > "release-1.11".
> > > > > >> >> > >
> > > > > >> >> > > *#### Core Source API Changes to backport (in REVERSE
> > order)*
> > > > > >> >> > >
> > > > > >> >> > >   (Use: "git log
> > > > > >> >> > >
> > > > flink-core/src/main/java/org/apache/flink/api/connector/source")
> > > > > >> >> > >
> > > > > >> >> > > commit 162c072e9265a7b6dd9d6f5459eb7974091c4c4e
> > > > > >> >> > > [FLINK-19492][core] Consolidate Source Events between
> > Source
> > > > API
> > > > > >> and
> > > > > >> >> > Split
> > > > > >> >> > > Reader API
> > > > > >> >> > >
> > > > > >> >> > > commit 428fbbad85379f71a9fbf1a1ab3e3dcfa9006436
> > > > > >> >> > > [FLINK-19205][core] Add access to configuration and
> > hostname
> > > in
> > > > > the
> > > > > >> >> > > SourceReaderContext
> > > > > >> >> > >
> > > > > >> >> > > commit 723e1790744ddc78d19c7c978442af1383f38d33
> > > > > >> >> > > [hotfix][core] Add to Source Enumerator convenience
> methods
> > > to
> > > > > >> assign
> > > > > >> >> > > single split
> > > > > >> >> > >
> > > > > >> >> > > *#### Connector Base Changes to Backport (in REVERSE
> > order)*
> > > > > >> >> > >
> > > > > >> >> > >   (Use: "git log flink-connectors/flink-connector-base")
> > > > > >> >> > >
> > > > > >> >> > > commit 401f56fe9d6b0271260edf9787cdcbfe4d03874d
> > > > > >> >> > > [FLINK-19427][FLINK-19489][tests] Fix test conditions for
> > > > > >> >> > > 'SplitFetcherTest.testNotifiesWhenGoingIdleConcurrent()'
> > > > > >> >> > >
> > > > > >> >> > > commit 68c5c2ff779d82a1ff81ffaf60d8a1b283797db1
> > > > > >> >> > > [FLINK-19448][connector base] Explicitly check for
> > > un-expected
> > > > > >> >> condition
> > > > > >> >> > > that would leave an inconsistent state
> > > > > >> >> > >
> > > > > >> >> > > commit 162c072e9265a7b6dd9d6f5459eb7974091c4c4e
> > > > > >> >> > > [FLINK-19492][core] Consolidate Source Events between
> > Source
> > > > API
> > > > > >> and
> > > > > >> >> > Split
> > > > > >> >> > > Reader API
> > > > > >> >> > >
> > > > > >> >> > > commit c1ca7a4c7c21ec8868c14cdddd43c559625b794c
> > > > > >> >> > > [refactor][tests] Move some source test utils from
> > > > > >> >> flink-connector-base
> > > > > >> >> > to
> > > > > >> >> > > flink-core
> > > > > >> >> > >
> > > > > >> >> > > commit ee5c4c211c35c70d28252363bbc8400453609977
> > > > > >> >> > > [FLINK-19251][connectors] Avoid confusing queue handling
> in
> > > > > >> >> > > "SplitReader.handleSplitsChanges()"
> > > > > >> >> > >
> > > > > >> >> > > commit 5abef56b2bf85bcac786f6b16b6899b6cced7176
> > > > > >> >> > > [FLINK-19250][connectors] Fix error propagation in
> > connector
> > > > base
> > > > > >> >> > > (SplitFetcherManager).
> > > > > >> >> > >
> > > > > >> >> > > commit 8fcca837c55a9216595ee4c03038b52747098dbb
> > > > > >> >> > > [hotfix][connectors] Improve JavaDocs for
> > > > > >> SingleThreadFetcherManager
> > > > > >> >> > >
> > > > > >> >> > > commit 4700bb5dde3303cbe98882f6beb7379425717b01
> > > > > >> >> > > [FLINK-19225][connectors] Various small improvements to
> > > > > >> >> SourceReaderBase
> > > > > >> >> > > (part 2)
> > > > > >> >> > >
> > > > > >> >> > > commit 12261c6b7ed6478a9b9f6a69cb58246b83cab9b7
> > > > > >> >> > > [FLINK-17393][connectors] (follow-up) Wakeup the
> > > SplitFetchers
> > > > > more
> > > > > >> >> > > elegantly.
> > > > > >> >> > >
> > > > > >> >> > > commit c60aaff0249bfd6b5871b7f82e03efc487a54d6b
> > > > > >> >> > > [hotfix][tests] Extend test coverage for
> > > > > >> FutureCompletingBlockingQueue
> > > > > >> >> > >
> > > > > >> >> > > commit cef8a587d7fd2fe64cc644da5ed095d82e46f631
> > > > > >> >> > > [FLINK-19245][connectors] Set default capacity for
> > > > > >> >> > > FutureCompletingBlockingQueue.
> > > > > >> >> > >
> > > > > >> >> > > commit 4ea95782b4c6a2538153d4d16ad3f4839c7de0fb
> > > > > >> >> > > [FLINK-19223][connectors] Simplify Availability Future
> > Model
> > > in
> > > > > >> Base
> > > > > >> >> > > Connector
> > > > > >> >> > >
> > > > > >> >> > > commit 511857049ba30c8ff0ee56da551fa4a479dc583e
> > > > > >> >> > > [FLINK-18128][connectors] Ensure idle split fetchers lead
> > to
> > > > > >> >> availability
> > > > > >> >> > > notifications.
> > > > > >> >> > >
> > > > > >> >> > > commit a8206467af0830dcb89623ea068b5ca3b3450c92
> > > > > >> >> > > [refactor][core] Eagerly initialize the FetchTask to
> > support
> > > > > proper
> > > > > >> >> unit
> > > > > >> >> > > testing
> > > > > >> >> > >
> > > > > >> >> > > commit 3b2f54bcb437f98e6137c904045cc51072b5c06b
> > > > > >> >> > > [hotfix][tests] Move constants in SplitFetcherTest
> relevant
> > > to
> > > > > only
> > > > > >> >> one
> > > > > >> >> > > test into test method
> > > > > >> >> > >
> > > > > >> >> > > commit d7625760a75a508bf05bcddc380bb4d62ee1743e
> > > > > >> >> > > [FLINK-19225][connectors] Various small improvements to
> > > > > >> >> SourceReaderBase
> > > > > >> >> > >
> > > > > >> >> > > commit a5b0d3297748c1be47ad579a88f24df2255a8df1
> > > > > >> >> > > [FLINK-17393][connectors] Wakeup the SplitFetchers more
> > > > > elegantly.
> > > > > >> >> > >
> > > > > >> >> > > commit f42a3ebc3e81a034b7221a803c153636fef34903
> > > > > >> >> > > [FLINK-18680][connectors] Make connector base
> > > > RecordsWithSplitIds
> > > > > >> more
> > > > > >> >> > > lightweight.
> > > > > >> >> > >
> > > > > >> >> > > commit e3d273de822b085183d09b275a445879ff94b350
> > > > > >> >> > > [FLINK-19162][connectors] Add 'recycle()' to the
> > > > > >> RecordsWithSplitIds
> > > > > >> >> to
> > > > > >> >> > > support reuse of heavy objects.
> > > > > >> >> > >
> > > > > >> >> > > commit 8ebc464c2520453a70001cd712abc8dee6ee89e0
> > > > > >> >> > > [hotfix][testing] Add a set of parameterizable testing
> > mocks
> > > > for
> > > > > >> the
> > > > > >> >> > Split
> > > > > >> >> > > Reader API
> > > > > >> >> > >
> > > > > >> >> > > commit 930a07438be1185388d7150640f294dfe2a5d378
> > > > > >> >> > > [hotfix][connectors] Add RequestSplitEvent to
> > > > > >> 'flink-connector-base'
> > > > > >> >> > >
> > > > > >> >> > > commit e945ce8a933bc378844782f784ca473c767ca159
> > > > > >> >> > > [refactor][connectors] Backport of the connector-base
> > > exception
> > > > > >> >> handling
> > > > > >> >> > > from the Kafka Connector Pull Request
> > > > > >> >> > >
> > > > > >> >> > >
> > > > > >> >> > > On Wed, Oct 28, 2020 at 5:49 PM Steven Wu <
> > > > stevenz3wu@gmail.com>
> > > > > >> >> wrote:
> > > > > >> >> > >
> > > > > >> >> > > > I would love to see this FLIP-27 source interface
> > > improvement
> > > > > [1]
> > > > > >> >> made
> > > > > >> >> > to
> > > > > >> >> > > > 1.11.3.
> > > > > >> >> > > >
> > > > > >> >> > > > [1] https://issues.apache.org/jira/browse/FLINK-19698
> > > > > >> >> > > >
> > > > > >> >> > > > On Wed, Oct 28, 2020 at 12:32 AM Tzu-Li (Gordon) Tai <
> > > > > >> >> > > tzulitai@apache.org>
> > > > > >> >> > > > wrote:
> > > > > >> >> > > >
> > > > > >> >> > > > > Thanks for the replies so far!
> > > > > >> >> > > > >
> > > > > >> >> > > > > Just to provide a brief update on the status of
> > blockers
> > > > for
> > > > > >> >> 1.11.3
> > > > > >> >> > so
> > > > > >> >> > > > far:
> > > > > >> >> > > > >
> > > > > >> >> > > > >
> > > > > >> >> > > > > *PR opened, pending reviewer*- [FLINK-19717]
> > > > > >> >> > SourceReaderBase.pollNext
> > > > > >> >> > > > may
> > > > > >> >> > > > > return END_OF_INPUT if SplitReader.fetch throws (
> > > > > >> >> > > > > https://github.com/apache/flink/pull/13776)
> > > > > >> >> > > > >
> > > > > >> >> > > > > *PR opened, reviewed + close to being merged*
> > > > > >> >> > > > > - [FLINK-19741] Timer service should skip restoring
> > from
> > > > raw
> > > > > >> keyed
> > > > > >> >> > > stream
> > > > > >> >> > > > > if it isn't the writer (
> > > > > >> >> https://github.com/apache/flink/pull/13761)
> > > > > >> >> > > > > - [FLINK-19748] Raw keyed stream key group iterator
> > > should
> > > > be
> > > > > >> >> > skipping
> > > > > >> >> > > > > unwritten key groups (
> > > > > >> https://github.com/apache/flink/pull/13772)
> > > > > >> >> > > > >
> > > > > >> >> > > > > *Merged*
> > > > > >> >> > > > > - [FLINK-19154] Application mode deletes HA data in
> > case
> > > of
> > > > > >> >> suspended
> > > > > >> >> > > > > ZooKeeper connection
> > > > > >> >> > > > > - [FLINK-19569] Upgrade ICU4J to 67.1+
> > > > > >> >> > > > >
> > > > > >> >> > > > > Right now as it seems, progress is mainly blocked on
> a
> > > > > reviewer
> > > > > >> >> for
> > > > > >> >> > > > > FLINK-19717.
> > > > > >> >> > > > > Meanwhile, Xintong is keeping an eye on test
> > > instabilities
> > > > > [1]
> > > > > >> to
> > > > > >> >> see
> > > > > >> >> > > if
> > > > > >> >> > > > > there are any fixes that should be applied to
> > > > `release-1.11`.
> > > > > >> >> > > > >
> > > > > >> >> > > > > This is also a reminder, that if there are other
> > blockers
> > > > > that
> > > > > >> we
> > > > > >> >> > need
> > > > > >> >> > > to
> > > > > >> >> > > > > be aware of, or a need to re-establish estimated time
> > for
> > > > > >> getting
> > > > > >> >> > fixes
> > > > > >> >> > > > in
> > > > > >> >> > > > > and delay the RC for 1.11.3, please do let us know!
> > > > > >> >> > > > >
> > > > > >> >> > > > > Cheers,
> > > > > >> >> > > > > Gordon
> > > > > >> >> > > > >
> > > > > >> >> > > > > [1]
> > > > > >> >> > > > >
> > > > > >> >> > > > >
> > > > > >> >> > > >
> > > > > >> >> > >
> > > > > >> >> >
> > > > > >> >>
> > > > > >>
> > > > >
> > > >
> > >
> >
> https://issues.apache.org/jira/issues/?jql=project%20%3D%20FLINK%20AND%20status%20in%20(Open%2C%20%22In%20Progress%22%2C%20Reopened)%20AND%20affectedVersion%20in%20(1.11.0%2C%201.11.1%2C%201.11.2)%20AND%20labels%20%3D%20test-stability
> > > > > >> >> > > > >
> > > > > >> >> > > > > On Mon, Oct 26, 2020 at 9:43 PM Kostas Kloudas <
> > > > > >> >> kkloudas@apache.org>
> > > > > >> >> > > > > wrote:
> > > > > >> >> > > > >
> > > > > >> >> > > > > > +1 for releasing Flink 1.11.3 as it contains a
> number
> > > of
> > > > > >> >> important
> > > > > >> >> > > > > > fixes and thanks Gordon and Xintong for
> volunteering.
> > > > > >> >> > > > > >
> > > > > >> >> > > > > > Cheers,
> > > > > >> >> > > > > > Kostas
> > > > > >> >> > > > > >
> > > > > >> >> > > > > > On Mon, Oct 26, 2020 at 4:37 AM Yu Li <
> > > carp84@gmail.com>
> > > > > >> wrote:
> > > > > >> >> > > > > > >
> > > > > >> >> > > > > > > +1 for releasing Flink 1.11.3, and thanks Gordon
> > and
> > > > > >> Xintong
> > > > > >> >> for
> > > > > >> >> > > > > > > volunteering as our release managers.
> > > > > >> >> > > > > > >
> > > > > >> >> > > > > > > Best Regards,
> > > > > >> >> > > > > > > Yu
> > > > > >> >> > > > > > >
> > > > > >> >> > > > > > >
> > > > > >> >> > > > > > > On Mon, 26 Oct 2020 at 09:45, Xintong Song <
> > > > > >> >> > tonysong820@gmail.com>
> > > > > >> >> > > > > > wrote:
> > > > > >> >> > > > > > >
> > > > > >> >> > > > > > > > Thanks Gordan for starting this discussion.
> > > > > >> >> > > > > > > > My pleasure to help with the release process.
> > > > > >> >> > > > > > > >
> > > > > >> >> > > > > > > > Thank you~
> > > > > >> >> > > > > > > >
> > > > > >> >> > > > > > > > Xintong Song
> > > > > >> >> > > > > > > >
> > > > > >> >> > > > > > > >
> > > > > >> >> > > > > > > >
> > > > > >> >> > > > > > > > On Fri, Oct 23, 2020 at 11:29 PM Till Rohrmann
> <
> > > > > >> >> > > > trohrmann@apache.org
> > > > > >> >> > > > > >
> > > > > >> >> > > > > > > > wrote:
> > > > > >> >> > > > > > > >
> > > > > >> >> > > > > > > > > Thanks for starting this discussion Gordon.
> > There
> > > > are
> > > > > >> over
> > > > > >> >> > 100
> > > > > >> >> > > > > issues
> > > > > >> >> > > > > > > > > which are fixed for 1.11.3. Hence +1 for a
> > > soonish
> > > > > >> 1.11.3
> > > > > >> >> > > > release.
> > > > > >> >> > > > > > Thanks
> > > > > >> >> > > > > > > > > for volunteering as our release managers
> Gordon
> > > and
> > > > > >> >> Xintong!
> > > > > >> >> > > > > > > > >
> > > > > >> >> > > > > > > > > Cheers,
> > > > > >> >> > > > > > > > > Till
> > > > > >> >> > > > > > > > >
> > > > > >> >> > > > > > > > > On Fri, Oct 23, 2020 at 5:02 PM Tzu-Li
> (Gordon)
> > > > Tai <
> > > > > >> >> > > > > > tzulitai@apache.org
> > > > > >> >> > > > > > > > >
> > > > > >> >> > > > > > > > > wrote:
> > > > > >> >> > > > > > > > >
> > > > > >> >> > > > > > > > >> Hi,
> > > > > >> >> > > > > > > > >>
> > > > > >> >> > > > > > > > >> Xintong and I would like to start a
> discussion
> > > for
> > > > > >> >> releasing
> > > > > >> >> > > > Flink
> > > > > >> >> > > > > > > > 1.11.3
> > > > > >> >> > > > > > > > >> soon.
> > > > > >> >> > > > > > > > >>
> > > > > >> >> > > > > > > > >> It seems like we already have a few pressing
> > > > issues
> > > > > >> that
> > > > > >> >> > needs
> > > > > >> >> > > > to
> > > > > >> >> > > > > be
> > > > > >> >> > > > > > > > >> included in a new hotfix release:
> > > > > >> >> > > > > > > > >>
> > > > > >> >> > > > > > > > >>    - Heap-based timers’ restore behaviour is
> > > > > causing a
> > > > > >> >> > > critical
> > > > > >> >> > > > > > recovery
> > > > > >> >> > > > > > > > >>    issue for StateFun [1] [2] [3].
> > > > > >> >> > > > > > > > >>    - There are several robustness issues for
> > the
> > > > > >> FLIP-27
> > > > > >> >> new
> > > > > >> >> > > > > source
> > > > > >> >> > > > > > API,
> > > > > >> >> > > > > > > > >>    such as [4]. We already have some users
> > using
> > > > the
> > > > > >> >> FLIP-27
> > > > > >> >> > > API
> > > > > >> >> > > > > > with
> > > > > >> >> > > > > > > > >> 1.11.x,
> > > > > >> >> > > > > > > > >>    so it would be important to get those
> fixes
> > > in
> > > > > for
> > > > > >> >> 1.11.x
> > > > > >> >> > > as
> > > > > >> >> > > > > > well.
> > > > > >> >> > > > > > > > >>
> > > > > >> >> > > > > > > > >> Apart from the issues that are already
> marked
> > as
> > > > > >> blocker
> > > > > >> >> for
> > > > > >> >> > > > > 1.11.3
> > > > > >> >> > > > > > in
> > > > > >> >> > > > > > > > our
> > > > > >> >> > > > > > > > >> JIRA [5], please let us know in this thread
> if
> > > > there
> > > > > >> is
> > > > > >> >> > > already
> > > > > >> >> > > > > > ongoing
> > > > > >> >> > > > > > > > >> work for other important fixes that we
> should
> > > try
> > > > to
> > > > > >> >> > include.
> > > > > >> >> > > > > > > > >>
> > > > > >> >> > > > > > > > >> Xintong and I would like to volunteer for
> > > managing
> > > > > >> this
> > > > > >> >> > > release,
> > > > > >> >> > > > > and
> > > > > >> >> > > > > > > > will
> > > > > >> >> > > > > > > > >> try to communicate the priority of pending
> > > > blockers
> > > > > >> over
> > > > > >> >> the
> > > > > >> >> > > > next
> > > > > >> >> > > > > > few
> > > > > >> >> > > > > > > > >> days.
> > > > > >> >> > > > > > > > >> Since the aforementioned issues are quite
> > > > critical,
> > > > > >> we’d
> > > > > >> >> > like
> > > > > >> >> > > to
> > > > > >> >> > > > > aim
> > > > > >> >> > > > > > > > >> for a *feature
> > > > > >> >> > > > > > > > >> freeze by the end of next week (Oct. 30th)*
> > and
> > > > > start
> > > > > >> the
> > > > > >> >> > > > release
> > > > > >> >> > > > > > voting
> > > > > >> >> > > > > > > > >> process the week after.
> > > > > >> >> > > > > > > > >> If that is too short of a notice and you
> might
> > > > need
> > > > > >> more
> > > > > >> >> > time,
> > > > > >> >> > > > > > please
> > > > > >> >> > > > > > > > let
> > > > > >> >> > > > > > > > >> us know!
> > > > > >> >> > > > > > > > >>
> > > > > >> >> > > > > > > > >> Cheers,
> > > > > >> >> > > > > > > > >> Gordon
> > > > > >> >> > > > > > > > >>
> > > > > >> >> > > > > > > > >> [1]
> > > > > https://issues.apache.org/jira/browse/FLINK-19692
> > > > > >> >> > > > > > > > >> [2]
> > > > > https://issues.apache.org/jira/browse/FLINK-19741
> > > > > >> >> > > > > > > > >> [3]
> > > > > https://issues.apache.org/jira/browse/FLINK-19748
> > > > > >> >> > > > > > > > >> [4]
> > > > > https://issues.apache.org/jira/browse/FLINK-19717
> > > > > >> >> > > > > > > > >> [5]
> > > > > >> >> > > > > > > > >>
> > > > > >> >> > > > > > > > >>
> > > > > >> >> > > > > > > >
> > > > > >> >> > > > > >
> > > > > >> >> > > > >
> > > > > >> >> > > >
> > > > > >> >> > >
> > > > > >> >> >
> > > > > >> >>
> > > > > >>
> > > > >
> > > >
> > >
> >
> https://issues.apache.org/jira/issues/?jql=project%20%3D%20FLINK%20AND%20priority%20%3D%20Blocker%20AND%20fixVersion%20%3D%201.11.3
> > > > > >> >> > > > > > > > >>
> > > > > >> >> > > > > > > > >
> > > > > >> >> > > > > > > >
> > > > > >> >> > > > > >
> > > > > >> >> > > > >
> > > > > >> >> > > >
> > > > > >> >> > >
> > > > > >> >> >
> > > > > >> >> >
> > > > > >> >> > --
> > > > > >> >> > Best, Jingsong Lee
> > > > > >> >> >
> > > > > >> >>
> > > > > >> >
> > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] Releasing Apache Flink 1.11.3

Posted by Steven Wu <st...@gmail.com>.
@Stephan Ewen <se...@apache.org> yeah, we can do that. don't worry about
it. your earlier email had the perfect explanation on why file source
shouldn't be backported.

On Tue, Nov 3, 2020 at 3:37 AM Stephan Ewen <se...@apache.org> wrote:

> @Steven would it be possible to initially copy some of the code into the
> iceberg source and later replace it by a dependency on the Flink file
> source?
>
> On Mon, Nov 2, 2020 at 8:33 PM Steven Wu <st...@gmail.com> wrote:
>
> > Stephan, thanks a lot for explaining the file connector. that makes
> sense.
> >
> > I was asking because we were trying to reuse some of the implementations
> in
> > the file source for Iceberg source. Flink Iceberg source lives in the
> > Iceberg repo, which is not possible to code against the master branch of
> > the Flink code.
> >
> > On Mon, Nov 2, 2020 at 3:31 AM Stephan Ewen <se...@apache.org> wrote:
> >
> > > Hi Steven!
> > >
> > > So far there are no plans to pick back the file system connector code.
> > This
> > > is still evolving and not finalized for 1.12, so I don't feel it is a
> > good
> > > candidate to be backported.
> > > However, with the base connector changes backported, you should be able
> > to
> > > run the file connector code from master against 1.11.3.
> > >
> > > The collect() utils can be picked back, I see no issue with that (it is
> > > isolated utilities).
> > >
> > > Best,
> > > Stephan
> > >
> > >
> > > On Mon, Nov 2, 2020 at 3:02 AM Steven Wu <st...@gmail.com> wrote:
> > >
> > > > Basically, it would be great to get the latest code in the
> > > > flink-connector-files (FLIP-27).
> > > >
> > > > On Sat, Oct 31, 2020 at 9:57 AM Steven Wu <st...@gmail.com>
> > wrote:
> > > >
> > > > > Stephan, it will be great if we can also backport the
> DataStreamUtils
> > > > > related commits that help with collecting output from unbounded
> > > streams.
> > > > > e.g.
> > > > >
> > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/flink/commit/09a7a66b7313fea64817fe960a8da1265b428efc
> > > > >
> > > > > I tried to copy and paste the code to unblock myself. but it
> quickly
> > > got
> > > > > into the rabbit hole of more and more code.
> > > > >
> > > > > On Fri, Oct 30, 2020 at 11:02 AM Stephan Ewen <se...@apache.org>
> > > wrote:
> > > > >
> > > > >> I have started with backporting the source API changes. Some minor
> > > > >> conflicts to solve, will need a bit more to finish this.
> > > > >>
> > > > >> On Fri, Oct 30, 2020 at 7:25 AM Tzu-Li (Gordon) Tai <
> > > > tzulitai@apache.org>
> > > > >> wrote:
> > > > >>
> > > > >> > @Stephan Ewen <se...@apache.org>
> > > > >> > Are there already plans or ongoing efforts for backporting the
> > list
> > > of
> > > > >> > FLIP-27 changes that you posted?
> > > > >> >
> > > > >> > On Thu, Oct 29, 2020 at 7:08 PM Xintong Song <
> > tonysong820@gmail.com
> > > >
> > > > >> > wrote:
> > > > >> >
> > > > >> >> Hi folks,
> > > > >> >>
> > > > >> >> Just to provide some updates concerning the status on the
> > > > >> >> test instabilities.
> > > > >> >>
> > > > >> >> Currently, we have 30 unresolved tickets labeled with `Affects
> > > > Version`
> > > > >> >> 1.11.x.
> > > > >> >>
> > > > >> >>
> > > > >>
> > > >
> > >
> >
> https://issues.apache.org/jira/browse/FLINK-19775?filter=12348580&jql=project%20%3D%20FLINK%20AND%20resolution%20%3D%20Unresolved%20AND%20affectedVersion%20in%20(1.11.0%2C%201.11.1%2C%201.11.2%2C%201.11.3)%20AND%20labels%20%3D%20test-stability%20ORDER%20BY%20created%20DESC
> > > > >> >>
> > > > >> >> Among the 30 tickets, 11 of them are:
> > > > >> >> - Have occured in the recent 3 months
> > > > >> >> - Not confirmed to be pure testability issues
> > > > >> >> - Not confirmed to be rare condition cases
> > > > >> >>
> > > > >> >> It would be nice if someone familiar with these components can
> > > take a
> > > > >> look
> > > > >> >> into these issues.
> > > > >> >>
> > > > >> >> - https://issues.apache.org/jira/browse/FLINK-17159 (ES6)
> > > > >> >> - https://issues.apache.org/jira/browse/FLINK-17912 (Kafka)
> > > > >> >> - https://issues.apache.org/jira/browse/FLINK-17949 (Kafka)
> > > > >> >> ⁃ https://issues.apache.org/jira/browse/FLINK-18444 (Kafka)
> > > > >> >> - https://issues.apache.org/jira/browse/FLINK-18634 (Kafka)
> > > > >> >> - https://issues.apache.org/jira/browse/FLINK-18648 (Kafka)
> > > > >> >> - https://issues.apache.org/jira/browse/FLINK-18807 (Kafka)
> > > > >> >> - https://issues.apache.org/jira/browse/FLINK-19369
> > > (BlobClientTest)
> > > > >> >> - https://issues.apache.org/jira/browse/FLINK-19436 (TPCDS)
> > > > >> >> - https://issues.apache.org/jira/browse/FLINK-19690
> > > (Format/Parquet)
> > > > >> >> - https://issues.apache.org/jira/browse/FLINK-19775
> > > > >> >> (SystemProcessingTimeServiceTest)
> > > > >> >>
> > > > >> >> Thank you~
> > > > >> >>
> > > > >> >> Xintong Song
> > > > >> >>
> > > > >> >>
> > > > >> >>
> > > > >> >> On Thu, Oct 29, 2020 at 10:21 AM Jingsong Li <
> > > jingsonglee0@gmail.com
> > > > >
> > > > >> >> wrote:
> > > > >> >>
> > > > >> >> > +1 to backport the FLIP-27 adjustments to 1.11.x.
> > > > >> >> >
> > > > >> >> > If possible, that would be great. Many people are looking
> > forward
> > > > to
> > > > >> the
> > > > >> >> > FLIP-27 interface, but they don't want to take the risk to
> > > upgrade
> > > > to
> > > > >> >> 1.12
> > > > >> >> > (And wait 1.12). After all, 1.11 is a relatively stable
> > version.
> > > > >> >> >
> > > > >> >> > Best,
> > > > >> >> > Jingsong
> > > > >> >> >
> > > > >> >> > On Thu, Oct 29, 2020 at 1:24 AM Stephan Ewen <
> sewen@apache.org
> > >
> > > > >> wrote:
> > > > >> >> >
> > > > >> >> > > Thanks for starting this.
> > > > >> >> > >
> > > > >> >> > > +1 form my side to backport the FLIP-27 adjustments to
> > 1.11.x.
> > > > >> >> > >
> > > > >> >> > > There were quite a few changes, and I think we need to
> > > > cherry-pick
> > > > >> >> them
> > > > >> >> > all
> > > > >> >> > > to not get some inconsistent mix of changes and many merge
> > > > >> conflicts.
> > > > >> >> > > I made a list below of what we need to add to
> "release-1.11".
> > > > >> >> > >
> > > > >> >> > > *#### Core Source API Changes to backport (in REVERSE
> order)*
> > > > >> >> > >
> > > > >> >> > >   (Use: "git log
> > > > >> >> > >
> > > flink-core/src/main/java/org/apache/flink/api/connector/source")
> > > > >> >> > >
> > > > >> >> > > commit 162c072e9265a7b6dd9d6f5459eb7974091c4c4e
> > > > >> >> > > [FLINK-19492][core] Consolidate Source Events between
> Source
> > > API
> > > > >> and
> > > > >> >> > Split
> > > > >> >> > > Reader API
> > > > >> >> > >
> > > > >> >> > > commit 428fbbad85379f71a9fbf1a1ab3e3dcfa9006436
> > > > >> >> > > [FLINK-19205][core] Add access to configuration and
> hostname
> > in
> > > > the
> > > > >> >> > > SourceReaderContext
> > > > >> >> > >
> > > > >> >> > > commit 723e1790744ddc78d19c7c978442af1383f38d33
> > > > >> >> > > [hotfix][core] Add to Source Enumerator convenience methods
> > to
> > > > >> assign
> > > > >> >> > > single split
> > > > >> >> > >
> > > > >> >> > > *#### Connector Base Changes to Backport (in REVERSE
> order)*
> > > > >> >> > >
> > > > >> >> > >   (Use: "git log flink-connectors/flink-connector-base")
> > > > >> >> > >
> > > > >> >> > > commit 401f56fe9d6b0271260edf9787cdcbfe4d03874d
> > > > >> >> > > [FLINK-19427][FLINK-19489][tests] Fix test conditions for
> > > > >> >> > > 'SplitFetcherTest.testNotifiesWhenGoingIdleConcurrent()'
> > > > >> >> > >
> > > > >> >> > > commit 68c5c2ff779d82a1ff81ffaf60d8a1b283797db1
> > > > >> >> > > [FLINK-19448][connector base] Explicitly check for
> > un-expected
> > > > >> >> condition
> > > > >> >> > > that would leave an inconsistent state
> > > > >> >> > >
> > > > >> >> > > commit 162c072e9265a7b6dd9d6f5459eb7974091c4c4e
> > > > >> >> > > [FLINK-19492][core] Consolidate Source Events between
> Source
> > > API
> > > > >> and
> > > > >> >> > Split
> > > > >> >> > > Reader API
> > > > >> >> > >
> > > > >> >> > > commit c1ca7a4c7c21ec8868c14cdddd43c559625b794c
> > > > >> >> > > [refactor][tests] Move some source test utils from
> > > > >> >> flink-connector-base
> > > > >> >> > to
> > > > >> >> > > flink-core
> > > > >> >> > >
> > > > >> >> > > commit ee5c4c211c35c70d28252363bbc8400453609977
> > > > >> >> > > [FLINK-19251][connectors] Avoid confusing queue handling in
> > > > >> >> > > "SplitReader.handleSplitsChanges()"
> > > > >> >> > >
> > > > >> >> > > commit 5abef56b2bf85bcac786f6b16b6899b6cced7176
> > > > >> >> > > [FLINK-19250][connectors] Fix error propagation in
> connector
> > > base
> > > > >> >> > > (SplitFetcherManager).
> > > > >> >> > >
> > > > >> >> > > commit 8fcca837c55a9216595ee4c03038b52747098dbb
> > > > >> >> > > [hotfix][connectors] Improve JavaDocs for
> > > > >> SingleThreadFetcherManager
> > > > >> >> > >
> > > > >> >> > > commit 4700bb5dde3303cbe98882f6beb7379425717b01
> > > > >> >> > > [FLINK-19225][connectors] Various small improvements to
> > > > >> >> SourceReaderBase
> > > > >> >> > > (part 2)
> > > > >> >> > >
> > > > >> >> > > commit 12261c6b7ed6478a9b9f6a69cb58246b83cab9b7
> > > > >> >> > > [FLINK-17393][connectors] (follow-up) Wakeup the
> > SplitFetchers
> > > > more
> > > > >> >> > > elegantly.
> > > > >> >> > >
> > > > >> >> > > commit c60aaff0249bfd6b5871b7f82e03efc487a54d6b
> > > > >> >> > > [hotfix][tests] Extend test coverage for
> > > > >> FutureCompletingBlockingQueue
> > > > >> >> > >
> > > > >> >> > > commit cef8a587d7fd2fe64cc644da5ed095d82e46f631
> > > > >> >> > > [FLINK-19245][connectors] Set default capacity for
> > > > >> >> > > FutureCompletingBlockingQueue.
> > > > >> >> > >
> > > > >> >> > > commit 4ea95782b4c6a2538153d4d16ad3f4839c7de0fb
> > > > >> >> > > [FLINK-19223][connectors] Simplify Availability Future
> Model
> > in
> > > > >> Base
> > > > >> >> > > Connector
> > > > >> >> > >
> > > > >> >> > > commit 511857049ba30c8ff0ee56da551fa4a479dc583e
> > > > >> >> > > [FLINK-18128][connectors] Ensure idle split fetchers lead
> to
> > > > >> >> availability
> > > > >> >> > > notifications.
> > > > >> >> > >
> > > > >> >> > > commit a8206467af0830dcb89623ea068b5ca3b3450c92
> > > > >> >> > > [refactor][core] Eagerly initialize the FetchTask to
> support
> > > > proper
> > > > >> >> unit
> > > > >> >> > > testing
> > > > >> >> > >
> > > > >> >> > > commit 3b2f54bcb437f98e6137c904045cc51072b5c06b
> > > > >> >> > > [hotfix][tests] Move constants in SplitFetcherTest relevant
> > to
> > > > only
> > > > >> >> one
> > > > >> >> > > test into test method
> > > > >> >> > >
> > > > >> >> > > commit d7625760a75a508bf05bcddc380bb4d62ee1743e
> > > > >> >> > > [FLINK-19225][connectors] Various small improvements to
> > > > >> >> SourceReaderBase
> > > > >> >> > >
> > > > >> >> > > commit a5b0d3297748c1be47ad579a88f24df2255a8df1
> > > > >> >> > > [FLINK-17393][connectors] Wakeup the SplitFetchers more
> > > > elegantly.
> > > > >> >> > >
> > > > >> >> > > commit f42a3ebc3e81a034b7221a803c153636fef34903
> > > > >> >> > > [FLINK-18680][connectors] Make connector base
> > > RecordsWithSplitIds
> > > > >> more
> > > > >> >> > > lightweight.
> > > > >> >> > >
> > > > >> >> > > commit e3d273de822b085183d09b275a445879ff94b350
> > > > >> >> > > [FLINK-19162][connectors] Add 'recycle()' to the
> > > > >> RecordsWithSplitIds
> > > > >> >> to
> > > > >> >> > > support reuse of heavy objects.
> > > > >> >> > >
> > > > >> >> > > commit 8ebc464c2520453a70001cd712abc8dee6ee89e0
> > > > >> >> > > [hotfix][testing] Add a set of parameterizable testing
> mocks
> > > for
> > > > >> the
> > > > >> >> > Split
> > > > >> >> > > Reader API
> > > > >> >> > >
> > > > >> >> > > commit 930a07438be1185388d7150640f294dfe2a5d378
> > > > >> >> > > [hotfix][connectors] Add RequestSplitEvent to
> > > > >> 'flink-connector-base'
> > > > >> >> > >
> > > > >> >> > > commit e945ce8a933bc378844782f784ca473c767ca159
> > > > >> >> > > [refactor][connectors] Backport of the connector-base
> > exception
> > > > >> >> handling
> > > > >> >> > > from the Kafka Connector Pull Request
> > > > >> >> > >
> > > > >> >> > >
> > > > >> >> > > On Wed, Oct 28, 2020 at 5:49 PM Steven Wu <
> > > stevenz3wu@gmail.com>
> > > > >> >> wrote:
> > > > >> >> > >
> > > > >> >> > > > I would love to see this FLIP-27 source interface
> > improvement
> > > > [1]
> > > > >> >> made
> > > > >> >> > to
> > > > >> >> > > > 1.11.3.
> > > > >> >> > > >
> > > > >> >> > > > [1] https://issues.apache.org/jira/browse/FLINK-19698
> > > > >> >> > > >
> > > > >> >> > > > On Wed, Oct 28, 2020 at 12:32 AM Tzu-Li (Gordon) Tai <
> > > > >> >> > > tzulitai@apache.org>
> > > > >> >> > > > wrote:
> > > > >> >> > > >
> > > > >> >> > > > > Thanks for the replies so far!
> > > > >> >> > > > >
> > > > >> >> > > > > Just to provide a brief update on the status of
> blockers
> > > for
> > > > >> >> 1.11.3
> > > > >> >> > so
> > > > >> >> > > > far:
> > > > >> >> > > > >
> > > > >> >> > > > >
> > > > >> >> > > > > *PR opened, pending reviewer*- [FLINK-19717]
> > > > >> >> > SourceReaderBase.pollNext
> > > > >> >> > > > may
> > > > >> >> > > > > return END_OF_INPUT if SplitReader.fetch throws (
> > > > >> >> > > > > https://github.com/apache/flink/pull/13776)
> > > > >> >> > > > >
> > > > >> >> > > > > *PR opened, reviewed + close to being merged*
> > > > >> >> > > > > - [FLINK-19741] Timer service should skip restoring
> from
> > > raw
> > > > >> keyed
> > > > >> >> > > stream
> > > > >> >> > > > > if it isn't the writer (
> > > > >> >> https://github.com/apache/flink/pull/13761)
> > > > >> >> > > > > - [FLINK-19748] Raw keyed stream key group iterator
> > should
> > > be
> > > > >> >> > skipping
> > > > >> >> > > > > unwritten key groups (
> > > > >> https://github.com/apache/flink/pull/13772)
> > > > >> >> > > > >
> > > > >> >> > > > > *Merged*
> > > > >> >> > > > > - [FLINK-19154] Application mode deletes HA data in
> case
> > of
> > > > >> >> suspended
> > > > >> >> > > > > ZooKeeper connection
> > > > >> >> > > > > - [FLINK-19569] Upgrade ICU4J to 67.1+
> > > > >> >> > > > >
> > > > >> >> > > > > Right now as it seems, progress is mainly blocked on a
> > > > reviewer
> > > > >> >> for
> > > > >> >> > > > > FLINK-19717.
> > > > >> >> > > > > Meanwhile, Xintong is keeping an eye on test
> > instabilities
> > > > [1]
> > > > >> to
> > > > >> >> see
> > > > >> >> > > if
> > > > >> >> > > > > there are any fixes that should be applied to
> > > `release-1.11`.
> > > > >> >> > > > >
> > > > >> >> > > > > This is also a reminder, that if there are other
> blockers
> > > > that
> > > > >> we
> > > > >> >> > need
> > > > >> >> > > to
> > > > >> >> > > > > be aware of, or a need to re-establish estimated time
> for
> > > > >> getting
> > > > >> >> > fixes
> > > > >> >> > > > in
> > > > >> >> > > > > and delay the RC for 1.11.3, please do let us know!
> > > > >> >> > > > >
> > > > >> >> > > > > Cheers,
> > > > >> >> > > > > Gordon
> > > > >> >> > > > >
> > > > >> >> > > > > [1]
> > > > >> >> > > > >
> > > > >> >> > > > >
> > > > >> >> > > >
> > > > >> >> > >
> > > > >> >> >
> > > > >> >>
> > > > >>
> > > >
> > >
> >
> https://issues.apache.org/jira/issues/?jql=project%20%3D%20FLINK%20AND%20status%20in%20(Open%2C%20%22In%20Progress%22%2C%20Reopened)%20AND%20affectedVersion%20in%20(1.11.0%2C%201.11.1%2C%201.11.2)%20AND%20labels%20%3D%20test-stability
> > > > >> >> > > > >
> > > > >> >> > > > > On Mon, Oct 26, 2020 at 9:43 PM Kostas Kloudas <
> > > > >> >> kkloudas@apache.org>
> > > > >> >> > > > > wrote:
> > > > >> >> > > > >
> > > > >> >> > > > > > +1 for releasing Flink 1.11.3 as it contains a number
> > of
> > > > >> >> important
> > > > >> >> > > > > > fixes and thanks Gordon and Xintong for volunteering.
> > > > >> >> > > > > >
> > > > >> >> > > > > > Cheers,
> > > > >> >> > > > > > Kostas
> > > > >> >> > > > > >
> > > > >> >> > > > > > On Mon, Oct 26, 2020 at 4:37 AM Yu Li <
> > carp84@gmail.com>
> > > > >> wrote:
> > > > >> >> > > > > > >
> > > > >> >> > > > > > > +1 for releasing Flink 1.11.3, and thanks Gordon
> and
> > > > >> Xintong
> > > > >> >> for
> > > > >> >> > > > > > > volunteering as our release managers.
> > > > >> >> > > > > > >
> > > > >> >> > > > > > > Best Regards,
> > > > >> >> > > > > > > Yu
> > > > >> >> > > > > > >
> > > > >> >> > > > > > >
> > > > >> >> > > > > > > On Mon, 26 Oct 2020 at 09:45, Xintong Song <
> > > > >> >> > tonysong820@gmail.com>
> > > > >> >> > > > > > wrote:
> > > > >> >> > > > > > >
> > > > >> >> > > > > > > > Thanks Gordan for starting this discussion.
> > > > >> >> > > > > > > > My pleasure to help with the release process.
> > > > >> >> > > > > > > >
> > > > >> >> > > > > > > > Thank you~
> > > > >> >> > > > > > > >
> > > > >> >> > > > > > > > Xintong Song
> > > > >> >> > > > > > > >
> > > > >> >> > > > > > > >
> > > > >> >> > > > > > > >
> > > > >> >> > > > > > > > On Fri, Oct 23, 2020 at 11:29 PM Till Rohrmann <
> > > > >> >> > > > trohrmann@apache.org
> > > > >> >> > > > > >
> > > > >> >> > > > > > > > wrote:
> > > > >> >> > > > > > > >
> > > > >> >> > > > > > > > > Thanks for starting this discussion Gordon.
> There
> > > are
> > > > >> over
> > > > >> >> > 100
> > > > >> >> > > > > issues
> > > > >> >> > > > > > > > > which are fixed for 1.11.3. Hence +1 for a
> > soonish
> > > > >> 1.11.3
> > > > >> >> > > > release.
> > > > >> >> > > > > > Thanks
> > > > >> >> > > > > > > > > for volunteering as our release managers Gordon
> > and
> > > > >> >> Xintong!
> > > > >> >> > > > > > > > >
> > > > >> >> > > > > > > > > Cheers,
> > > > >> >> > > > > > > > > Till
> > > > >> >> > > > > > > > >
> > > > >> >> > > > > > > > > On Fri, Oct 23, 2020 at 5:02 PM Tzu-Li (Gordon)
> > > Tai <
> > > > >> >> > > > > > tzulitai@apache.org
> > > > >> >> > > > > > > > >
> > > > >> >> > > > > > > > > wrote:
> > > > >> >> > > > > > > > >
> > > > >> >> > > > > > > > >> Hi,
> > > > >> >> > > > > > > > >>
> > > > >> >> > > > > > > > >> Xintong and I would like to start a discussion
> > for
> > > > >> >> releasing
> > > > >> >> > > > Flink
> > > > >> >> > > > > > > > 1.11.3
> > > > >> >> > > > > > > > >> soon.
> > > > >> >> > > > > > > > >>
> > > > >> >> > > > > > > > >> It seems like we already have a few pressing
> > > issues
> > > > >> that
> > > > >> >> > needs
> > > > >> >> > > > to
> > > > >> >> > > > > be
> > > > >> >> > > > > > > > >> included in a new hotfix release:
> > > > >> >> > > > > > > > >>
> > > > >> >> > > > > > > > >>    - Heap-based timers’ restore behaviour is
> > > > causing a
> > > > >> >> > > critical
> > > > >> >> > > > > > recovery
> > > > >> >> > > > > > > > >>    issue for StateFun [1] [2] [3].
> > > > >> >> > > > > > > > >>    - There are several robustness issues for
> the
> > > > >> FLIP-27
> > > > >> >> new
> > > > >> >> > > > > source
> > > > >> >> > > > > > API,
> > > > >> >> > > > > > > > >>    such as [4]. We already have some users
> using
> > > the
> > > > >> >> FLIP-27
> > > > >> >> > > API
> > > > >> >> > > > > > with
> > > > >> >> > > > > > > > >> 1.11.x,
> > > > >> >> > > > > > > > >>    so it would be important to get those fixes
> > in
> > > > for
> > > > >> >> 1.11.x
> > > > >> >> > > as
> > > > >> >> > > > > > well.
> > > > >> >> > > > > > > > >>
> > > > >> >> > > > > > > > >> Apart from the issues that are already marked
> as
> > > > >> blocker
> > > > >> >> for
> > > > >> >> > > > > 1.11.3
> > > > >> >> > > > > > in
> > > > >> >> > > > > > > > our
> > > > >> >> > > > > > > > >> JIRA [5], please let us know in this thread if
> > > there
> > > > >> is
> > > > >> >> > > already
> > > > >> >> > > > > > ongoing
> > > > >> >> > > > > > > > >> work for other important fixes that we should
> > try
> > > to
> > > > >> >> > include.
> > > > >> >> > > > > > > > >>
> > > > >> >> > > > > > > > >> Xintong and I would like to volunteer for
> > managing
> > > > >> this
> > > > >> >> > > release,
> > > > >> >> > > > > and
> > > > >> >> > > > > > > > will
> > > > >> >> > > > > > > > >> try to communicate the priority of pending
> > > blockers
> > > > >> over
> > > > >> >> the
> > > > >> >> > > > next
> > > > >> >> > > > > > few
> > > > >> >> > > > > > > > >> days.
> > > > >> >> > > > > > > > >> Since the aforementioned issues are quite
> > > critical,
> > > > >> we’d
> > > > >> >> > like
> > > > >> >> > > to
> > > > >> >> > > > > aim
> > > > >> >> > > > > > > > >> for a *feature
> > > > >> >> > > > > > > > >> freeze by the end of next week (Oct. 30th)*
> and
> > > > start
> > > > >> the
> > > > >> >> > > > release
> > > > >> >> > > > > > voting
> > > > >> >> > > > > > > > >> process the week after.
> > > > >> >> > > > > > > > >> If that is too short of a notice and you might
> > > need
> > > > >> more
> > > > >> >> > time,
> > > > >> >> > > > > > please
> > > > >> >> > > > > > > > let
> > > > >> >> > > > > > > > >> us know!
> > > > >> >> > > > > > > > >>
> > > > >> >> > > > > > > > >> Cheers,
> > > > >> >> > > > > > > > >> Gordon
> > > > >> >> > > > > > > > >>
> > > > >> >> > > > > > > > >> [1]
> > > > https://issues.apache.org/jira/browse/FLINK-19692
> > > > >> >> > > > > > > > >> [2]
> > > > https://issues.apache.org/jira/browse/FLINK-19741
> > > > >> >> > > > > > > > >> [3]
> > > > https://issues.apache.org/jira/browse/FLINK-19748
> > > > >> >> > > > > > > > >> [4]
> > > > https://issues.apache.org/jira/browse/FLINK-19717
> > > > >> >> > > > > > > > >> [5]
> > > > >> >> > > > > > > > >>
> > > > >> >> > > > > > > > >>
> > > > >> >> > > > > > > >
> > > > >> >> > > > > >
> > > > >> >> > > > >
> > > > >> >> > > >
> > > > >> >> > >
> > > > >> >> >
> > > > >> >>
> > > > >>
> > > >
> > >
> >
> https://issues.apache.org/jira/issues/?jql=project%20%3D%20FLINK%20AND%20priority%20%3D%20Blocker%20AND%20fixVersion%20%3D%201.11.3
> > > > >> >> > > > > > > > >>
> > > > >> >> > > > > > > > >
> > > > >> >> > > > > > > >
> > > > >> >> > > > > >
> > > > >> >> > > > >
> > > > >> >> > > >
> > > > >> >> > >
> > > > >> >> >
> > > > >> >> >
> > > > >> >> > --
> > > > >> >> > Best, Jingsong Lee
> > > > >> >> >
> > > > >> >>
> > > > >> >
> > > > >>
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] Releasing Apache Flink 1.11.3

Posted by Stephan Ewen <se...@apache.org>.
@Steven would it be possible to initially copy some of the code into the
iceberg source and later replace it by a dependency on the Flink file
source?

On Mon, Nov 2, 2020 at 8:33 PM Steven Wu <st...@gmail.com> wrote:

> Stephan, thanks a lot for explaining the file connector. that makes sense.
>
> I was asking because we were trying to reuse some of the implementations in
> the file source for Iceberg source. Flink Iceberg source lives in the
> Iceberg repo, which is not possible to code against the master branch of
> the Flink code.
>
> On Mon, Nov 2, 2020 at 3:31 AM Stephan Ewen <se...@apache.org> wrote:
>
> > Hi Steven!
> >
> > So far there are no plans to pick back the file system connector code.
> This
> > is still evolving and not finalized for 1.12, so I don't feel it is a
> good
> > candidate to be backported.
> > However, with the base connector changes backported, you should be able
> to
> > run the file connector code from master against 1.11.3.
> >
> > The collect() utils can be picked back, I see no issue with that (it is
> > isolated utilities).
> >
> > Best,
> > Stephan
> >
> >
> > On Mon, Nov 2, 2020 at 3:02 AM Steven Wu <st...@gmail.com> wrote:
> >
> > > Basically, it would be great to get the latest code in the
> > > flink-connector-files (FLIP-27).
> > >
> > > On Sat, Oct 31, 2020 at 9:57 AM Steven Wu <st...@gmail.com>
> wrote:
> > >
> > > > Stephan, it will be great if we can also backport the DataStreamUtils
> > > > related commits that help with collecting output from unbounded
> > streams.
> > > > e.g.
> > > >
> > > >
> > > >
> > >
> >
> https://github.com/apache/flink/commit/09a7a66b7313fea64817fe960a8da1265b428efc
> > > >
> > > > I tried to copy and paste the code to unblock myself. but it quickly
> > got
> > > > into the rabbit hole of more and more code.
> > > >
> > > > On Fri, Oct 30, 2020 at 11:02 AM Stephan Ewen <se...@apache.org>
> > wrote:
> > > >
> > > >> I have started with backporting the source API changes. Some minor
> > > >> conflicts to solve, will need a bit more to finish this.
> > > >>
> > > >> On Fri, Oct 30, 2020 at 7:25 AM Tzu-Li (Gordon) Tai <
> > > tzulitai@apache.org>
> > > >> wrote:
> > > >>
> > > >> > @Stephan Ewen <se...@apache.org>
> > > >> > Are there already plans or ongoing efforts for backporting the
> list
> > of
> > > >> > FLIP-27 changes that you posted?
> > > >> >
> > > >> > On Thu, Oct 29, 2020 at 7:08 PM Xintong Song <
> tonysong820@gmail.com
> > >
> > > >> > wrote:
> > > >> >
> > > >> >> Hi folks,
> > > >> >>
> > > >> >> Just to provide some updates concerning the status on the
> > > >> >> test instabilities.
> > > >> >>
> > > >> >> Currently, we have 30 unresolved tickets labeled with `Affects
> > > Version`
> > > >> >> 1.11.x.
> > > >> >>
> > > >> >>
> > > >>
> > >
> >
> https://issues.apache.org/jira/browse/FLINK-19775?filter=12348580&jql=project%20%3D%20FLINK%20AND%20resolution%20%3D%20Unresolved%20AND%20affectedVersion%20in%20(1.11.0%2C%201.11.1%2C%201.11.2%2C%201.11.3)%20AND%20labels%20%3D%20test-stability%20ORDER%20BY%20created%20DESC
> > > >> >>
> > > >> >> Among the 30 tickets, 11 of them are:
> > > >> >> - Have occured in the recent 3 months
> > > >> >> - Not confirmed to be pure testability issues
> > > >> >> - Not confirmed to be rare condition cases
> > > >> >>
> > > >> >> It would be nice if someone familiar with these components can
> > take a
> > > >> look
> > > >> >> into these issues.
> > > >> >>
> > > >> >> - https://issues.apache.org/jira/browse/FLINK-17159 (ES6)
> > > >> >> - https://issues.apache.org/jira/browse/FLINK-17912 (Kafka)
> > > >> >> - https://issues.apache.org/jira/browse/FLINK-17949 (Kafka)
> > > >> >> ⁃ https://issues.apache.org/jira/browse/FLINK-18444 (Kafka)
> > > >> >> - https://issues.apache.org/jira/browse/FLINK-18634 (Kafka)
> > > >> >> - https://issues.apache.org/jira/browse/FLINK-18648 (Kafka)
> > > >> >> - https://issues.apache.org/jira/browse/FLINK-18807 (Kafka)
> > > >> >> - https://issues.apache.org/jira/browse/FLINK-19369
> > (BlobClientTest)
> > > >> >> - https://issues.apache.org/jira/browse/FLINK-19436 (TPCDS)
> > > >> >> - https://issues.apache.org/jira/browse/FLINK-19690
> > (Format/Parquet)
> > > >> >> - https://issues.apache.org/jira/browse/FLINK-19775
> > > >> >> (SystemProcessingTimeServiceTest)
> > > >> >>
> > > >> >> Thank you~
> > > >> >>
> > > >> >> Xintong Song
> > > >> >>
> > > >> >>
> > > >> >>
> > > >> >> On Thu, Oct 29, 2020 at 10:21 AM Jingsong Li <
> > jingsonglee0@gmail.com
> > > >
> > > >> >> wrote:
> > > >> >>
> > > >> >> > +1 to backport the FLIP-27 adjustments to 1.11.x.
> > > >> >> >
> > > >> >> > If possible, that would be great. Many people are looking
> forward
> > > to
> > > >> the
> > > >> >> > FLIP-27 interface, but they don't want to take the risk to
> > upgrade
> > > to
> > > >> >> 1.12
> > > >> >> > (And wait 1.12). After all, 1.11 is a relatively stable
> version.
> > > >> >> >
> > > >> >> > Best,
> > > >> >> > Jingsong
> > > >> >> >
> > > >> >> > On Thu, Oct 29, 2020 at 1:24 AM Stephan Ewen <sewen@apache.org
> >
> > > >> wrote:
> > > >> >> >
> > > >> >> > > Thanks for starting this.
> > > >> >> > >
> > > >> >> > > +1 form my side to backport the FLIP-27 adjustments to
> 1.11.x.
> > > >> >> > >
> > > >> >> > > There were quite a few changes, and I think we need to
> > > cherry-pick
> > > >> >> them
> > > >> >> > all
> > > >> >> > > to not get some inconsistent mix of changes and many merge
> > > >> conflicts.
> > > >> >> > > I made a list below of what we need to add to "release-1.11".
> > > >> >> > >
> > > >> >> > > *#### Core Source API Changes to backport (in REVERSE order)*
> > > >> >> > >
> > > >> >> > >   (Use: "git log
> > > >> >> > >
> > flink-core/src/main/java/org/apache/flink/api/connector/source")
> > > >> >> > >
> > > >> >> > > commit 162c072e9265a7b6dd9d6f5459eb7974091c4c4e
> > > >> >> > > [FLINK-19492][core] Consolidate Source Events between Source
> > API
> > > >> and
> > > >> >> > Split
> > > >> >> > > Reader API
> > > >> >> > >
> > > >> >> > > commit 428fbbad85379f71a9fbf1a1ab3e3dcfa9006436
> > > >> >> > > [FLINK-19205][core] Add access to configuration and hostname
> in
> > > the
> > > >> >> > > SourceReaderContext
> > > >> >> > >
> > > >> >> > > commit 723e1790744ddc78d19c7c978442af1383f38d33
> > > >> >> > > [hotfix][core] Add to Source Enumerator convenience methods
> to
> > > >> assign
> > > >> >> > > single split
> > > >> >> > >
> > > >> >> > > *#### Connector Base Changes to Backport (in REVERSE order)*
> > > >> >> > >
> > > >> >> > >   (Use: "git log flink-connectors/flink-connector-base")
> > > >> >> > >
> > > >> >> > > commit 401f56fe9d6b0271260edf9787cdcbfe4d03874d
> > > >> >> > > [FLINK-19427][FLINK-19489][tests] Fix test conditions for
> > > >> >> > > 'SplitFetcherTest.testNotifiesWhenGoingIdleConcurrent()'
> > > >> >> > >
> > > >> >> > > commit 68c5c2ff779d82a1ff81ffaf60d8a1b283797db1
> > > >> >> > > [FLINK-19448][connector base] Explicitly check for
> un-expected
> > > >> >> condition
> > > >> >> > > that would leave an inconsistent state
> > > >> >> > >
> > > >> >> > > commit 162c072e9265a7b6dd9d6f5459eb7974091c4c4e
> > > >> >> > > [FLINK-19492][core] Consolidate Source Events between Source
> > API
> > > >> and
> > > >> >> > Split
> > > >> >> > > Reader API
> > > >> >> > >
> > > >> >> > > commit c1ca7a4c7c21ec8868c14cdddd43c559625b794c
> > > >> >> > > [refactor][tests] Move some source test utils from
> > > >> >> flink-connector-base
> > > >> >> > to
> > > >> >> > > flink-core
> > > >> >> > >
> > > >> >> > > commit ee5c4c211c35c70d28252363bbc8400453609977
> > > >> >> > > [FLINK-19251][connectors] Avoid confusing queue handling in
> > > >> >> > > "SplitReader.handleSplitsChanges()"
> > > >> >> > >
> > > >> >> > > commit 5abef56b2bf85bcac786f6b16b6899b6cced7176
> > > >> >> > > [FLINK-19250][connectors] Fix error propagation in connector
> > base
> > > >> >> > > (SplitFetcherManager).
> > > >> >> > >
> > > >> >> > > commit 8fcca837c55a9216595ee4c03038b52747098dbb
> > > >> >> > > [hotfix][connectors] Improve JavaDocs for
> > > >> SingleThreadFetcherManager
> > > >> >> > >
> > > >> >> > > commit 4700bb5dde3303cbe98882f6beb7379425717b01
> > > >> >> > > [FLINK-19225][connectors] Various small improvements to
> > > >> >> SourceReaderBase
> > > >> >> > > (part 2)
> > > >> >> > >
> > > >> >> > > commit 12261c6b7ed6478a9b9f6a69cb58246b83cab9b7
> > > >> >> > > [FLINK-17393][connectors] (follow-up) Wakeup the
> SplitFetchers
> > > more
> > > >> >> > > elegantly.
> > > >> >> > >
> > > >> >> > > commit c60aaff0249bfd6b5871b7f82e03efc487a54d6b
> > > >> >> > > [hotfix][tests] Extend test coverage for
> > > >> FutureCompletingBlockingQueue
> > > >> >> > >
> > > >> >> > > commit cef8a587d7fd2fe64cc644da5ed095d82e46f631
> > > >> >> > > [FLINK-19245][connectors] Set default capacity for
> > > >> >> > > FutureCompletingBlockingQueue.
> > > >> >> > >
> > > >> >> > > commit 4ea95782b4c6a2538153d4d16ad3f4839c7de0fb
> > > >> >> > > [FLINK-19223][connectors] Simplify Availability Future Model
> in
> > > >> Base
> > > >> >> > > Connector
> > > >> >> > >
> > > >> >> > > commit 511857049ba30c8ff0ee56da551fa4a479dc583e
> > > >> >> > > [FLINK-18128][connectors] Ensure idle split fetchers lead to
> > > >> >> availability
> > > >> >> > > notifications.
> > > >> >> > >
> > > >> >> > > commit a8206467af0830dcb89623ea068b5ca3b3450c92
> > > >> >> > > [refactor][core] Eagerly initialize the FetchTask to support
> > > proper
> > > >> >> unit
> > > >> >> > > testing
> > > >> >> > >
> > > >> >> > > commit 3b2f54bcb437f98e6137c904045cc51072b5c06b
> > > >> >> > > [hotfix][tests] Move constants in SplitFetcherTest relevant
> to
> > > only
> > > >> >> one
> > > >> >> > > test into test method
> > > >> >> > >
> > > >> >> > > commit d7625760a75a508bf05bcddc380bb4d62ee1743e
> > > >> >> > > [FLINK-19225][connectors] Various small improvements to
> > > >> >> SourceReaderBase
> > > >> >> > >
> > > >> >> > > commit a5b0d3297748c1be47ad579a88f24df2255a8df1
> > > >> >> > > [FLINK-17393][connectors] Wakeup the SplitFetchers more
> > > elegantly.
> > > >> >> > >
> > > >> >> > > commit f42a3ebc3e81a034b7221a803c153636fef34903
> > > >> >> > > [FLINK-18680][connectors] Make connector base
> > RecordsWithSplitIds
> > > >> more
> > > >> >> > > lightweight.
> > > >> >> > >
> > > >> >> > > commit e3d273de822b085183d09b275a445879ff94b350
> > > >> >> > > [FLINK-19162][connectors] Add 'recycle()' to the
> > > >> RecordsWithSplitIds
> > > >> >> to
> > > >> >> > > support reuse of heavy objects.
> > > >> >> > >
> > > >> >> > > commit 8ebc464c2520453a70001cd712abc8dee6ee89e0
> > > >> >> > > [hotfix][testing] Add a set of parameterizable testing mocks
> > for
> > > >> the
> > > >> >> > Split
> > > >> >> > > Reader API
> > > >> >> > >
> > > >> >> > > commit 930a07438be1185388d7150640f294dfe2a5d378
> > > >> >> > > [hotfix][connectors] Add RequestSplitEvent to
> > > >> 'flink-connector-base'
> > > >> >> > >
> > > >> >> > > commit e945ce8a933bc378844782f784ca473c767ca159
> > > >> >> > > [refactor][connectors] Backport of the connector-base
> exception
> > > >> >> handling
> > > >> >> > > from the Kafka Connector Pull Request
> > > >> >> > >
> > > >> >> > >
> > > >> >> > > On Wed, Oct 28, 2020 at 5:49 PM Steven Wu <
> > stevenz3wu@gmail.com>
> > > >> >> wrote:
> > > >> >> > >
> > > >> >> > > > I would love to see this FLIP-27 source interface
> improvement
> > > [1]
> > > >> >> made
> > > >> >> > to
> > > >> >> > > > 1.11.3.
> > > >> >> > > >
> > > >> >> > > > [1] https://issues.apache.org/jira/browse/FLINK-19698
> > > >> >> > > >
> > > >> >> > > > On Wed, Oct 28, 2020 at 12:32 AM Tzu-Li (Gordon) Tai <
> > > >> >> > > tzulitai@apache.org>
> > > >> >> > > > wrote:
> > > >> >> > > >
> > > >> >> > > > > Thanks for the replies so far!
> > > >> >> > > > >
> > > >> >> > > > > Just to provide a brief update on the status of blockers
> > for
> > > >> >> 1.11.3
> > > >> >> > so
> > > >> >> > > > far:
> > > >> >> > > > >
> > > >> >> > > > >
> > > >> >> > > > > *PR opened, pending reviewer*- [FLINK-19717]
> > > >> >> > SourceReaderBase.pollNext
> > > >> >> > > > may
> > > >> >> > > > > return END_OF_INPUT if SplitReader.fetch throws (
> > > >> >> > > > > https://github.com/apache/flink/pull/13776)
> > > >> >> > > > >
> > > >> >> > > > > *PR opened, reviewed + close to being merged*
> > > >> >> > > > > - [FLINK-19741] Timer service should skip restoring from
> > raw
> > > >> keyed
> > > >> >> > > stream
> > > >> >> > > > > if it isn't the writer (
> > > >> >> https://github.com/apache/flink/pull/13761)
> > > >> >> > > > > - [FLINK-19748] Raw keyed stream key group iterator
> should
> > be
> > > >> >> > skipping
> > > >> >> > > > > unwritten key groups (
> > > >> https://github.com/apache/flink/pull/13772)
> > > >> >> > > > >
> > > >> >> > > > > *Merged*
> > > >> >> > > > > - [FLINK-19154] Application mode deletes HA data in case
> of
> > > >> >> suspended
> > > >> >> > > > > ZooKeeper connection
> > > >> >> > > > > - [FLINK-19569] Upgrade ICU4J to 67.1+
> > > >> >> > > > >
> > > >> >> > > > > Right now as it seems, progress is mainly blocked on a
> > > reviewer
> > > >> >> for
> > > >> >> > > > > FLINK-19717.
> > > >> >> > > > > Meanwhile, Xintong is keeping an eye on test
> instabilities
> > > [1]
> > > >> to
> > > >> >> see
> > > >> >> > > if
> > > >> >> > > > > there are any fixes that should be applied to
> > `release-1.11`.
> > > >> >> > > > >
> > > >> >> > > > > This is also a reminder, that if there are other blockers
> > > that
> > > >> we
> > > >> >> > need
> > > >> >> > > to
> > > >> >> > > > > be aware of, or a need to re-establish estimated time for
> > > >> getting
> > > >> >> > fixes
> > > >> >> > > > in
> > > >> >> > > > > and delay the RC for 1.11.3, please do let us know!
> > > >> >> > > > >
> > > >> >> > > > > Cheers,
> > > >> >> > > > > Gordon
> > > >> >> > > > >
> > > >> >> > > > > [1]
> > > >> >> > > > >
> > > >> >> > > > >
> > > >> >> > > >
> > > >> >> > >
> > > >> >> >
> > > >> >>
> > > >>
> > >
> >
> https://issues.apache.org/jira/issues/?jql=project%20%3D%20FLINK%20AND%20status%20in%20(Open%2C%20%22In%20Progress%22%2C%20Reopened)%20AND%20affectedVersion%20in%20(1.11.0%2C%201.11.1%2C%201.11.2)%20AND%20labels%20%3D%20test-stability
> > > >> >> > > > >
> > > >> >> > > > > On Mon, Oct 26, 2020 at 9:43 PM Kostas Kloudas <
> > > >> >> kkloudas@apache.org>
> > > >> >> > > > > wrote:
> > > >> >> > > > >
> > > >> >> > > > > > +1 for releasing Flink 1.11.3 as it contains a number
> of
> > > >> >> important
> > > >> >> > > > > > fixes and thanks Gordon and Xintong for volunteering.
> > > >> >> > > > > >
> > > >> >> > > > > > Cheers,
> > > >> >> > > > > > Kostas
> > > >> >> > > > > >
> > > >> >> > > > > > On Mon, Oct 26, 2020 at 4:37 AM Yu Li <
> carp84@gmail.com>
> > > >> wrote:
> > > >> >> > > > > > >
> > > >> >> > > > > > > +1 for releasing Flink 1.11.3, and thanks Gordon and
> > > >> Xintong
> > > >> >> for
> > > >> >> > > > > > > volunteering as our release managers.
> > > >> >> > > > > > >
> > > >> >> > > > > > > Best Regards,
> > > >> >> > > > > > > Yu
> > > >> >> > > > > > >
> > > >> >> > > > > > >
> > > >> >> > > > > > > On Mon, 26 Oct 2020 at 09:45, Xintong Song <
> > > >> >> > tonysong820@gmail.com>
> > > >> >> > > > > > wrote:
> > > >> >> > > > > > >
> > > >> >> > > > > > > > Thanks Gordan for starting this discussion.
> > > >> >> > > > > > > > My pleasure to help with the release process.
> > > >> >> > > > > > > >
> > > >> >> > > > > > > > Thank you~
> > > >> >> > > > > > > >
> > > >> >> > > > > > > > Xintong Song
> > > >> >> > > > > > > >
> > > >> >> > > > > > > >
> > > >> >> > > > > > > >
> > > >> >> > > > > > > > On Fri, Oct 23, 2020 at 11:29 PM Till Rohrmann <
> > > >> >> > > > trohrmann@apache.org
> > > >> >> > > > > >
> > > >> >> > > > > > > > wrote:
> > > >> >> > > > > > > >
> > > >> >> > > > > > > > > Thanks for starting this discussion Gordon. There
> > are
> > > >> over
> > > >> >> > 100
> > > >> >> > > > > issues
> > > >> >> > > > > > > > > which are fixed for 1.11.3. Hence +1 for a
> soonish
> > > >> 1.11.3
> > > >> >> > > > release.
> > > >> >> > > > > > Thanks
> > > >> >> > > > > > > > > for volunteering as our release managers Gordon
> and
> > > >> >> Xintong!
> > > >> >> > > > > > > > >
> > > >> >> > > > > > > > > Cheers,
> > > >> >> > > > > > > > > Till
> > > >> >> > > > > > > > >
> > > >> >> > > > > > > > > On Fri, Oct 23, 2020 at 5:02 PM Tzu-Li (Gordon)
> > Tai <
> > > >> >> > > > > > tzulitai@apache.org
> > > >> >> > > > > > > > >
> > > >> >> > > > > > > > > wrote:
> > > >> >> > > > > > > > >
> > > >> >> > > > > > > > >> Hi,
> > > >> >> > > > > > > > >>
> > > >> >> > > > > > > > >> Xintong and I would like to start a discussion
> for
> > > >> >> releasing
> > > >> >> > > > Flink
> > > >> >> > > > > > > > 1.11.3
> > > >> >> > > > > > > > >> soon.
> > > >> >> > > > > > > > >>
> > > >> >> > > > > > > > >> It seems like we already have a few pressing
> > issues
> > > >> that
> > > >> >> > needs
> > > >> >> > > > to
> > > >> >> > > > > be
> > > >> >> > > > > > > > >> included in a new hotfix release:
> > > >> >> > > > > > > > >>
> > > >> >> > > > > > > > >>    - Heap-based timers’ restore behaviour is
> > > causing a
> > > >> >> > > critical
> > > >> >> > > > > > recovery
> > > >> >> > > > > > > > >>    issue for StateFun [1] [2] [3].
> > > >> >> > > > > > > > >>    - There are several robustness issues for the
> > > >> FLIP-27
> > > >> >> new
> > > >> >> > > > > source
> > > >> >> > > > > > API,
> > > >> >> > > > > > > > >>    such as [4]. We already have some users using
> > the
> > > >> >> FLIP-27
> > > >> >> > > API
> > > >> >> > > > > > with
> > > >> >> > > > > > > > >> 1.11.x,
> > > >> >> > > > > > > > >>    so it would be important to get those fixes
> in
> > > for
> > > >> >> 1.11.x
> > > >> >> > > as
> > > >> >> > > > > > well.
> > > >> >> > > > > > > > >>
> > > >> >> > > > > > > > >> Apart from the issues that are already marked as
> > > >> blocker
> > > >> >> for
> > > >> >> > > > > 1.11.3
> > > >> >> > > > > > in
> > > >> >> > > > > > > > our
> > > >> >> > > > > > > > >> JIRA [5], please let us know in this thread if
> > there
> > > >> is
> > > >> >> > > already
> > > >> >> > > > > > ongoing
> > > >> >> > > > > > > > >> work for other important fixes that we should
> try
> > to
> > > >> >> > include.
> > > >> >> > > > > > > > >>
> > > >> >> > > > > > > > >> Xintong and I would like to volunteer for
> managing
> > > >> this
> > > >> >> > > release,
> > > >> >> > > > > and
> > > >> >> > > > > > > > will
> > > >> >> > > > > > > > >> try to communicate the priority of pending
> > blockers
> > > >> over
> > > >> >> the
> > > >> >> > > > next
> > > >> >> > > > > > few
> > > >> >> > > > > > > > >> days.
> > > >> >> > > > > > > > >> Since the aforementioned issues are quite
> > critical,
> > > >> we’d
> > > >> >> > like
> > > >> >> > > to
> > > >> >> > > > > aim
> > > >> >> > > > > > > > >> for a *feature
> > > >> >> > > > > > > > >> freeze by the end of next week (Oct. 30th)* and
> > > start
> > > >> the
> > > >> >> > > > release
> > > >> >> > > > > > voting
> > > >> >> > > > > > > > >> process the week after.
> > > >> >> > > > > > > > >> If that is too short of a notice and you might
> > need
> > > >> more
> > > >> >> > time,
> > > >> >> > > > > > please
> > > >> >> > > > > > > > let
> > > >> >> > > > > > > > >> us know!
> > > >> >> > > > > > > > >>
> > > >> >> > > > > > > > >> Cheers,
> > > >> >> > > > > > > > >> Gordon
> > > >> >> > > > > > > > >>
> > > >> >> > > > > > > > >> [1]
> > > https://issues.apache.org/jira/browse/FLINK-19692
> > > >> >> > > > > > > > >> [2]
> > > https://issues.apache.org/jira/browse/FLINK-19741
> > > >> >> > > > > > > > >> [3]
> > > https://issues.apache.org/jira/browse/FLINK-19748
> > > >> >> > > > > > > > >> [4]
> > > https://issues.apache.org/jira/browse/FLINK-19717
> > > >> >> > > > > > > > >> [5]
> > > >> >> > > > > > > > >>
> > > >> >> > > > > > > > >>
> > > >> >> > > > > > > >
> > > >> >> > > > > >
> > > >> >> > > > >
> > > >> >> > > >
> > > >> >> > >
> > > >> >> >
> > > >> >>
> > > >>
> > >
> >
> https://issues.apache.org/jira/issues/?jql=project%20%3D%20FLINK%20AND%20priority%20%3D%20Blocker%20AND%20fixVersion%20%3D%201.11.3
> > > >> >> > > > > > > > >>
> > > >> >> > > > > > > > >
> > > >> >> > > > > > > >
> > > >> >> > > > > >
> > > >> >> > > > >
> > > >> >> > > >
> > > >> >> > >
> > > >> >> >
> > > >> >> >
> > > >> >> > --
> > > >> >> > Best, Jingsong Lee
> > > >> >> >
> > > >> >>
> > > >> >
> > > >>
> > > >
> > >
> >
>

Re: [DISCUSS] Releasing Apache Flink 1.11.3

Posted by Steven Wu <st...@gmail.com>.
Stephan, thanks a lot for explaining the file connector. that makes sense.

I was asking because we were trying to reuse some of the implementations in
the file source for Iceberg source. Flink Iceberg source lives in the
Iceberg repo, which is not possible to code against the master branch of
the Flink code.

On Mon, Nov 2, 2020 at 3:31 AM Stephan Ewen <se...@apache.org> wrote:

> Hi Steven!
>
> So far there are no plans to pick back the file system connector code. This
> is still evolving and not finalized for 1.12, so I don't feel it is a good
> candidate to be backported.
> However, with the base connector changes backported, you should be able to
> run the file connector code from master against 1.11.3.
>
> The collect() utils can be picked back, I see no issue with that (it is
> isolated utilities).
>
> Best,
> Stephan
>
>
> On Mon, Nov 2, 2020 at 3:02 AM Steven Wu <st...@gmail.com> wrote:
>
> > Basically, it would be great to get the latest code in the
> > flink-connector-files (FLIP-27).
> >
> > On Sat, Oct 31, 2020 at 9:57 AM Steven Wu <st...@gmail.com> wrote:
> >
> > > Stephan, it will be great if we can also backport the DataStreamUtils
> > > related commits that help with collecting output from unbounded
> streams.
> > > e.g.
> > >
> > >
> > >
> >
> https://github.com/apache/flink/commit/09a7a66b7313fea64817fe960a8da1265b428efc
> > >
> > > I tried to copy and paste the code to unblock myself. but it quickly
> got
> > > into the rabbit hole of more and more code.
> > >
> > > On Fri, Oct 30, 2020 at 11:02 AM Stephan Ewen <se...@apache.org>
> wrote:
> > >
> > >> I have started with backporting the source API changes. Some minor
> > >> conflicts to solve, will need a bit more to finish this.
> > >>
> > >> On Fri, Oct 30, 2020 at 7:25 AM Tzu-Li (Gordon) Tai <
> > tzulitai@apache.org>
> > >> wrote:
> > >>
> > >> > @Stephan Ewen <se...@apache.org>
> > >> > Are there already plans or ongoing efforts for backporting the list
> of
> > >> > FLIP-27 changes that you posted?
> > >> >
> > >> > On Thu, Oct 29, 2020 at 7:08 PM Xintong Song <tonysong820@gmail.com
> >
> > >> > wrote:
> > >> >
> > >> >> Hi folks,
> > >> >>
> > >> >> Just to provide some updates concerning the status on the
> > >> >> test instabilities.
> > >> >>
> > >> >> Currently, we have 30 unresolved tickets labeled with `Affects
> > Version`
> > >> >> 1.11.x.
> > >> >>
> > >> >>
> > >>
> >
> https://issues.apache.org/jira/browse/FLINK-19775?filter=12348580&jql=project%20%3D%20FLINK%20AND%20resolution%20%3D%20Unresolved%20AND%20affectedVersion%20in%20(1.11.0%2C%201.11.1%2C%201.11.2%2C%201.11.3)%20AND%20labels%20%3D%20test-stability%20ORDER%20BY%20created%20DESC
> > >> >>
> > >> >> Among the 30 tickets, 11 of them are:
> > >> >> - Have occured in the recent 3 months
> > >> >> - Not confirmed to be pure testability issues
> > >> >> - Not confirmed to be rare condition cases
> > >> >>
> > >> >> It would be nice if someone familiar with these components can
> take a
> > >> look
> > >> >> into these issues.
> > >> >>
> > >> >> - https://issues.apache.org/jira/browse/FLINK-17159 (ES6)
> > >> >> - https://issues.apache.org/jira/browse/FLINK-17912 (Kafka)
> > >> >> - https://issues.apache.org/jira/browse/FLINK-17949 (Kafka)
> > >> >> ⁃ https://issues.apache.org/jira/browse/FLINK-18444 (Kafka)
> > >> >> - https://issues.apache.org/jira/browse/FLINK-18634 (Kafka)
> > >> >> - https://issues.apache.org/jira/browse/FLINK-18648 (Kafka)
> > >> >> - https://issues.apache.org/jira/browse/FLINK-18807 (Kafka)
> > >> >> - https://issues.apache.org/jira/browse/FLINK-19369
> (BlobClientTest)
> > >> >> - https://issues.apache.org/jira/browse/FLINK-19436 (TPCDS)
> > >> >> - https://issues.apache.org/jira/browse/FLINK-19690
> (Format/Parquet)
> > >> >> - https://issues.apache.org/jira/browse/FLINK-19775
> > >> >> (SystemProcessingTimeServiceTest)
> > >> >>
> > >> >> Thank you~
> > >> >>
> > >> >> Xintong Song
> > >> >>
> > >> >>
> > >> >>
> > >> >> On Thu, Oct 29, 2020 at 10:21 AM Jingsong Li <
> jingsonglee0@gmail.com
> > >
> > >> >> wrote:
> > >> >>
> > >> >> > +1 to backport the FLIP-27 adjustments to 1.11.x.
> > >> >> >
> > >> >> > If possible, that would be great. Many people are looking forward
> > to
> > >> the
> > >> >> > FLIP-27 interface, but they don't want to take the risk to
> upgrade
> > to
> > >> >> 1.12
> > >> >> > (And wait 1.12). After all, 1.11 is a relatively stable version.
> > >> >> >
> > >> >> > Best,
> > >> >> > Jingsong
> > >> >> >
> > >> >> > On Thu, Oct 29, 2020 at 1:24 AM Stephan Ewen <se...@apache.org>
> > >> wrote:
> > >> >> >
> > >> >> > > Thanks for starting this.
> > >> >> > >
> > >> >> > > +1 form my side to backport the FLIP-27 adjustments to 1.11.x.
> > >> >> > >
> > >> >> > > There were quite a few changes, and I think we need to
> > cherry-pick
> > >> >> them
> > >> >> > all
> > >> >> > > to not get some inconsistent mix of changes and many merge
> > >> conflicts.
> > >> >> > > I made a list below of what we need to add to "release-1.11".
> > >> >> > >
> > >> >> > > *#### Core Source API Changes to backport (in REVERSE order)*
> > >> >> > >
> > >> >> > >   (Use: "git log
> > >> >> > >
> flink-core/src/main/java/org/apache/flink/api/connector/source")
> > >> >> > >
> > >> >> > > commit 162c072e9265a7b6dd9d6f5459eb7974091c4c4e
> > >> >> > > [FLINK-19492][core] Consolidate Source Events between Source
> API
> > >> and
> > >> >> > Split
> > >> >> > > Reader API
> > >> >> > >
> > >> >> > > commit 428fbbad85379f71a9fbf1a1ab3e3dcfa9006436
> > >> >> > > [FLINK-19205][core] Add access to configuration and hostname in
> > the
> > >> >> > > SourceReaderContext
> > >> >> > >
> > >> >> > > commit 723e1790744ddc78d19c7c978442af1383f38d33
> > >> >> > > [hotfix][core] Add to Source Enumerator convenience methods to
> > >> assign
> > >> >> > > single split
> > >> >> > >
> > >> >> > > *#### Connector Base Changes to Backport (in REVERSE order)*
> > >> >> > >
> > >> >> > >   (Use: "git log flink-connectors/flink-connector-base")
> > >> >> > >
> > >> >> > > commit 401f56fe9d6b0271260edf9787cdcbfe4d03874d
> > >> >> > > [FLINK-19427][FLINK-19489][tests] Fix test conditions for
> > >> >> > > 'SplitFetcherTest.testNotifiesWhenGoingIdleConcurrent()'
> > >> >> > >
> > >> >> > > commit 68c5c2ff779d82a1ff81ffaf60d8a1b283797db1
> > >> >> > > [FLINK-19448][connector base] Explicitly check for un-expected
> > >> >> condition
> > >> >> > > that would leave an inconsistent state
> > >> >> > >
> > >> >> > > commit 162c072e9265a7b6dd9d6f5459eb7974091c4c4e
> > >> >> > > [FLINK-19492][core] Consolidate Source Events between Source
> API
> > >> and
> > >> >> > Split
> > >> >> > > Reader API
> > >> >> > >
> > >> >> > > commit c1ca7a4c7c21ec8868c14cdddd43c559625b794c
> > >> >> > > [refactor][tests] Move some source test utils from
> > >> >> flink-connector-base
> > >> >> > to
> > >> >> > > flink-core
> > >> >> > >
> > >> >> > > commit ee5c4c211c35c70d28252363bbc8400453609977
> > >> >> > > [FLINK-19251][connectors] Avoid confusing queue handling in
> > >> >> > > "SplitReader.handleSplitsChanges()"
> > >> >> > >
> > >> >> > > commit 5abef56b2bf85bcac786f6b16b6899b6cced7176
> > >> >> > > [FLINK-19250][connectors] Fix error propagation in connector
> base
> > >> >> > > (SplitFetcherManager).
> > >> >> > >
> > >> >> > > commit 8fcca837c55a9216595ee4c03038b52747098dbb
> > >> >> > > [hotfix][connectors] Improve JavaDocs for
> > >> SingleThreadFetcherManager
> > >> >> > >
> > >> >> > > commit 4700bb5dde3303cbe98882f6beb7379425717b01
> > >> >> > > [FLINK-19225][connectors] Various small improvements to
> > >> >> SourceReaderBase
> > >> >> > > (part 2)
> > >> >> > >
> > >> >> > > commit 12261c6b7ed6478a9b9f6a69cb58246b83cab9b7
> > >> >> > > [FLINK-17393][connectors] (follow-up) Wakeup the SplitFetchers
> > more
> > >> >> > > elegantly.
> > >> >> > >
> > >> >> > > commit c60aaff0249bfd6b5871b7f82e03efc487a54d6b
> > >> >> > > [hotfix][tests] Extend test coverage for
> > >> FutureCompletingBlockingQueue
> > >> >> > >
> > >> >> > > commit cef8a587d7fd2fe64cc644da5ed095d82e46f631
> > >> >> > > [FLINK-19245][connectors] Set default capacity for
> > >> >> > > FutureCompletingBlockingQueue.
> > >> >> > >
> > >> >> > > commit 4ea95782b4c6a2538153d4d16ad3f4839c7de0fb
> > >> >> > > [FLINK-19223][connectors] Simplify Availability Future Model in
> > >> Base
> > >> >> > > Connector
> > >> >> > >
> > >> >> > > commit 511857049ba30c8ff0ee56da551fa4a479dc583e
> > >> >> > > [FLINK-18128][connectors] Ensure idle split fetchers lead to
> > >> >> availability
> > >> >> > > notifications.
> > >> >> > >
> > >> >> > > commit a8206467af0830dcb89623ea068b5ca3b3450c92
> > >> >> > > [refactor][core] Eagerly initialize the FetchTask to support
> > proper
> > >> >> unit
> > >> >> > > testing
> > >> >> > >
> > >> >> > > commit 3b2f54bcb437f98e6137c904045cc51072b5c06b
> > >> >> > > [hotfix][tests] Move constants in SplitFetcherTest relevant to
> > only
> > >> >> one
> > >> >> > > test into test method
> > >> >> > >
> > >> >> > > commit d7625760a75a508bf05bcddc380bb4d62ee1743e
> > >> >> > > [FLINK-19225][connectors] Various small improvements to
> > >> >> SourceReaderBase
> > >> >> > >
> > >> >> > > commit a5b0d3297748c1be47ad579a88f24df2255a8df1
> > >> >> > > [FLINK-17393][connectors] Wakeup the SplitFetchers more
> > elegantly.
> > >> >> > >
> > >> >> > > commit f42a3ebc3e81a034b7221a803c153636fef34903
> > >> >> > > [FLINK-18680][connectors] Make connector base
> RecordsWithSplitIds
> > >> more
> > >> >> > > lightweight.
> > >> >> > >
> > >> >> > > commit e3d273de822b085183d09b275a445879ff94b350
> > >> >> > > [FLINK-19162][connectors] Add 'recycle()' to the
> > >> RecordsWithSplitIds
> > >> >> to
> > >> >> > > support reuse of heavy objects.
> > >> >> > >
> > >> >> > > commit 8ebc464c2520453a70001cd712abc8dee6ee89e0
> > >> >> > > [hotfix][testing] Add a set of parameterizable testing mocks
> for
> > >> the
> > >> >> > Split
> > >> >> > > Reader API
> > >> >> > >
> > >> >> > > commit 930a07438be1185388d7150640f294dfe2a5d378
> > >> >> > > [hotfix][connectors] Add RequestSplitEvent to
> > >> 'flink-connector-base'
> > >> >> > >
> > >> >> > > commit e945ce8a933bc378844782f784ca473c767ca159
> > >> >> > > [refactor][connectors] Backport of the connector-base exception
> > >> >> handling
> > >> >> > > from the Kafka Connector Pull Request
> > >> >> > >
> > >> >> > >
> > >> >> > > On Wed, Oct 28, 2020 at 5:49 PM Steven Wu <
> stevenz3wu@gmail.com>
> > >> >> wrote:
> > >> >> > >
> > >> >> > > > I would love to see this FLIP-27 source interface improvement
> > [1]
> > >> >> made
> > >> >> > to
> > >> >> > > > 1.11.3.
> > >> >> > > >
> > >> >> > > > [1] https://issues.apache.org/jira/browse/FLINK-19698
> > >> >> > > >
> > >> >> > > > On Wed, Oct 28, 2020 at 12:32 AM Tzu-Li (Gordon) Tai <
> > >> >> > > tzulitai@apache.org>
> > >> >> > > > wrote:
> > >> >> > > >
> > >> >> > > > > Thanks for the replies so far!
> > >> >> > > > >
> > >> >> > > > > Just to provide a brief update on the status of blockers
> for
> > >> >> 1.11.3
> > >> >> > so
> > >> >> > > > far:
> > >> >> > > > >
> > >> >> > > > >
> > >> >> > > > > *PR opened, pending reviewer*- [FLINK-19717]
> > >> >> > SourceReaderBase.pollNext
> > >> >> > > > may
> > >> >> > > > > return END_OF_INPUT if SplitReader.fetch throws (
> > >> >> > > > > https://github.com/apache/flink/pull/13776)
> > >> >> > > > >
> > >> >> > > > > *PR opened, reviewed + close to being merged*
> > >> >> > > > > - [FLINK-19741] Timer service should skip restoring from
> raw
> > >> keyed
> > >> >> > > stream
> > >> >> > > > > if it isn't the writer (
> > >> >> https://github.com/apache/flink/pull/13761)
> > >> >> > > > > - [FLINK-19748] Raw keyed stream key group iterator should
> be
> > >> >> > skipping
> > >> >> > > > > unwritten key groups (
> > >> https://github.com/apache/flink/pull/13772)
> > >> >> > > > >
> > >> >> > > > > *Merged*
> > >> >> > > > > - [FLINK-19154] Application mode deletes HA data in case of
> > >> >> suspended
> > >> >> > > > > ZooKeeper connection
> > >> >> > > > > - [FLINK-19569] Upgrade ICU4J to 67.1+
> > >> >> > > > >
> > >> >> > > > > Right now as it seems, progress is mainly blocked on a
> > reviewer
> > >> >> for
> > >> >> > > > > FLINK-19717.
> > >> >> > > > > Meanwhile, Xintong is keeping an eye on test instabilities
> > [1]
> > >> to
> > >> >> see
> > >> >> > > if
> > >> >> > > > > there are any fixes that should be applied to
> `release-1.11`.
> > >> >> > > > >
> > >> >> > > > > This is also a reminder, that if there are other blockers
> > that
> > >> we
> > >> >> > need
> > >> >> > > to
> > >> >> > > > > be aware of, or a need to re-establish estimated time for
> > >> getting
> > >> >> > fixes
> > >> >> > > > in
> > >> >> > > > > and delay the RC for 1.11.3, please do let us know!
> > >> >> > > > >
> > >> >> > > > > Cheers,
> > >> >> > > > > Gordon
> > >> >> > > > >
> > >> >> > > > > [1]
> > >> >> > > > >
> > >> >> > > > >
> > >> >> > > >
> > >> >> > >
> > >> >> >
> > >> >>
> > >>
> >
> https://issues.apache.org/jira/issues/?jql=project%20%3D%20FLINK%20AND%20status%20in%20(Open%2C%20%22In%20Progress%22%2C%20Reopened)%20AND%20affectedVersion%20in%20(1.11.0%2C%201.11.1%2C%201.11.2)%20AND%20labels%20%3D%20test-stability
> > >> >> > > > >
> > >> >> > > > > On Mon, Oct 26, 2020 at 9:43 PM Kostas Kloudas <
> > >> >> kkloudas@apache.org>
> > >> >> > > > > wrote:
> > >> >> > > > >
> > >> >> > > > > > +1 for releasing Flink 1.11.3 as it contains a number of
> > >> >> important
> > >> >> > > > > > fixes and thanks Gordon and Xintong for volunteering.
> > >> >> > > > > >
> > >> >> > > > > > Cheers,
> > >> >> > > > > > Kostas
> > >> >> > > > > >
> > >> >> > > > > > On Mon, Oct 26, 2020 at 4:37 AM Yu Li <ca...@gmail.com>
> > >> wrote:
> > >> >> > > > > > >
> > >> >> > > > > > > +1 for releasing Flink 1.11.3, and thanks Gordon and
> > >> Xintong
> > >> >> for
> > >> >> > > > > > > volunteering as our release managers.
> > >> >> > > > > > >
> > >> >> > > > > > > Best Regards,
> > >> >> > > > > > > Yu
> > >> >> > > > > > >
> > >> >> > > > > > >
> > >> >> > > > > > > On Mon, 26 Oct 2020 at 09:45, Xintong Song <
> > >> >> > tonysong820@gmail.com>
> > >> >> > > > > > wrote:
> > >> >> > > > > > >
> > >> >> > > > > > > > Thanks Gordan for starting this discussion.
> > >> >> > > > > > > > My pleasure to help with the release process.
> > >> >> > > > > > > >
> > >> >> > > > > > > > Thank you~
> > >> >> > > > > > > >
> > >> >> > > > > > > > Xintong Song
> > >> >> > > > > > > >
> > >> >> > > > > > > >
> > >> >> > > > > > > >
> > >> >> > > > > > > > On Fri, Oct 23, 2020 at 11:29 PM Till Rohrmann <
> > >> >> > > > trohrmann@apache.org
> > >> >> > > > > >
> > >> >> > > > > > > > wrote:
> > >> >> > > > > > > >
> > >> >> > > > > > > > > Thanks for starting this discussion Gordon. There
> are
> > >> over
> > >> >> > 100
> > >> >> > > > > issues
> > >> >> > > > > > > > > which are fixed for 1.11.3. Hence +1 for a soonish
> > >> 1.11.3
> > >> >> > > > release.
> > >> >> > > > > > Thanks
> > >> >> > > > > > > > > for volunteering as our release managers Gordon and
> > >> >> Xintong!
> > >> >> > > > > > > > >
> > >> >> > > > > > > > > Cheers,
> > >> >> > > > > > > > > Till
> > >> >> > > > > > > > >
> > >> >> > > > > > > > > On Fri, Oct 23, 2020 at 5:02 PM Tzu-Li (Gordon)
> Tai <
> > >> >> > > > > > tzulitai@apache.org
> > >> >> > > > > > > > >
> > >> >> > > > > > > > > wrote:
> > >> >> > > > > > > > >
> > >> >> > > > > > > > >> Hi,
> > >> >> > > > > > > > >>
> > >> >> > > > > > > > >> Xintong and I would like to start a discussion for
> > >> >> releasing
> > >> >> > > > Flink
> > >> >> > > > > > > > 1.11.3
> > >> >> > > > > > > > >> soon.
> > >> >> > > > > > > > >>
> > >> >> > > > > > > > >> It seems like we already have a few pressing
> issues
> > >> that
> > >> >> > needs
> > >> >> > > > to
> > >> >> > > > > be
> > >> >> > > > > > > > >> included in a new hotfix release:
> > >> >> > > > > > > > >>
> > >> >> > > > > > > > >>    - Heap-based timers’ restore behaviour is
> > causing a
> > >> >> > > critical
> > >> >> > > > > > recovery
> > >> >> > > > > > > > >>    issue for StateFun [1] [2] [3].
> > >> >> > > > > > > > >>    - There are several robustness issues for the
> > >> FLIP-27
> > >> >> new
> > >> >> > > > > source
> > >> >> > > > > > API,
> > >> >> > > > > > > > >>    such as [4]. We already have some users using
> the
> > >> >> FLIP-27
> > >> >> > > API
> > >> >> > > > > > with
> > >> >> > > > > > > > >> 1.11.x,
> > >> >> > > > > > > > >>    so it would be important to get those fixes in
> > for
> > >> >> 1.11.x
> > >> >> > > as
> > >> >> > > > > > well.
> > >> >> > > > > > > > >>
> > >> >> > > > > > > > >> Apart from the issues that are already marked as
> > >> blocker
> > >> >> for
> > >> >> > > > > 1.11.3
> > >> >> > > > > > in
> > >> >> > > > > > > > our
> > >> >> > > > > > > > >> JIRA [5], please let us know in this thread if
> there
> > >> is
> > >> >> > > already
> > >> >> > > > > > ongoing
> > >> >> > > > > > > > >> work for other important fixes that we should try
> to
> > >> >> > include.
> > >> >> > > > > > > > >>
> > >> >> > > > > > > > >> Xintong and I would like to volunteer for managing
> > >> this
> > >> >> > > release,
> > >> >> > > > > and
> > >> >> > > > > > > > will
> > >> >> > > > > > > > >> try to communicate the priority of pending
> blockers
> > >> over
> > >> >> the
> > >> >> > > > next
> > >> >> > > > > > few
> > >> >> > > > > > > > >> days.
> > >> >> > > > > > > > >> Since the aforementioned issues are quite
> critical,
> > >> we’d
> > >> >> > like
> > >> >> > > to
> > >> >> > > > > aim
> > >> >> > > > > > > > >> for a *feature
> > >> >> > > > > > > > >> freeze by the end of next week (Oct. 30th)* and
> > start
> > >> the
> > >> >> > > > release
> > >> >> > > > > > voting
> > >> >> > > > > > > > >> process the week after.
> > >> >> > > > > > > > >> If that is too short of a notice and you might
> need
> > >> more
> > >> >> > time,
> > >> >> > > > > > please
> > >> >> > > > > > > > let
> > >> >> > > > > > > > >> us know!
> > >> >> > > > > > > > >>
> > >> >> > > > > > > > >> Cheers,
> > >> >> > > > > > > > >> Gordon
> > >> >> > > > > > > > >>
> > >> >> > > > > > > > >> [1]
> > https://issues.apache.org/jira/browse/FLINK-19692
> > >> >> > > > > > > > >> [2]
> > https://issues.apache.org/jira/browse/FLINK-19741
> > >> >> > > > > > > > >> [3]
> > https://issues.apache.org/jira/browse/FLINK-19748
> > >> >> > > > > > > > >> [4]
> > https://issues.apache.org/jira/browse/FLINK-19717
> > >> >> > > > > > > > >> [5]
> > >> >> > > > > > > > >>
> > >> >> > > > > > > > >>
> > >> >> > > > > > > >
> > >> >> > > > > >
> > >> >> > > > >
> > >> >> > > >
> > >> >> > >
> > >> >> >
> > >> >>
> > >>
> >
> https://issues.apache.org/jira/issues/?jql=project%20%3D%20FLINK%20AND%20priority%20%3D%20Blocker%20AND%20fixVersion%20%3D%201.11.3
> > >> >> > > > > > > > >>
> > >> >> > > > > > > > >
> > >> >> > > > > > > >
> > >> >> > > > > >
> > >> >> > > > >
> > >> >> > > >
> > >> >> > >
> > >> >> >
> > >> >> >
> > >> >> > --
> > >> >> > Best, Jingsong Lee
> > >> >> >
> > >> >>
> > >> >
> > >>
> > >
> >
>

Re: [DISCUSS] Releasing Apache Flink 1.11.3

Posted by Stephan Ewen <se...@apache.org>.
Hi Steven!

So far there are no plans to pick back the file system connector code. This
is still evolving and not finalized for 1.12, so I don't feel it is a good
candidate to be backported.
However, with the base connector changes backported, you should be able to
run the file connector code from master against 1.11.3.

The collect() utils can be picked back, I see no issue with that (it is
isolated utilities).

Best,
Stephan


On Mon, Nov 2, 2020 at 3:02 AM Steven Wu <st...@gmail.com> wrote:

> Basically, it would be great to get the latest code in the
> flink-connector-files (FLIP-27).
>
> On Sat, Oct 31, 2020 at 9:57 AM Steven Wu <st...@gmail.com> wrote:
>
> > Stephan, it will be great if we can also backport the DataStreamUtils
> > related commits that help with collecting output from unbounded streams.
> > e.g.
> >
> >
> >
> https://github.com/apache/flink/commit/09a7a66b7313fea64817fe960a8da1265b428efc
> >
> > I tried to copy and paste the code to unblock myself. but it quickly got
> > into the rabbit hole of more and more code.
> >
> > On Fri, Oct 30, 2020 at 11:02 AM Stephan Ewen <se...@apache.org> wrote:
> >
> >> I have started with backporting the source API changes. Some minor
> >> conflicts to solve, will need a bit more to finish this.
> >>
> >> On Fri, Oct 30, 2020 at 7:25 AM Tzu-Li (Gordon) Tai <
> tzulitai@apache.org>
> >> wrote:
> >>
> >> > @Stephan Ewen <se...@apache.org>
> >> > Are there already plans or ongoing efforts for backporting the list of
> >> > FLIP-27 changes that you posted?
> >> >
> >> > On Thu, Oct 29, 2020 at 7:08 PM Xintong Song <to...@gmail.com>
> >> > wrote:
> >> >
> >> >> Hi folks,
> >> >>
> >> >> Just to provide some updates concerning the status on the
> >> >> test instabilities.
> >> >>
> >> >> Currently, we have 30 unresolved tickets labeled with `Affects
> Version`
> >> >> 1.11.x.
> >> >>
> >> >>
> >>
> https://issues.apache.org/jira/browse/FLINK-19775?filter=12348580&jql=project%20%3D%20FLINK%20AND%20resolution%20%3D%20Unresolved%20AND%20affectedVersion%20in%20(1.11.0%2C%201.11.1%2C%201.11.2%2C%201.11.3)%20AND%20labels%20%3D%20test-stability%20ORDER%20BY%20created%20DESC
> >> >>
> >> >> Among the 30 tickets, 11 of them are:
> >> >> - Have occured in the recent 3 months
> >> >> - Not confirmed to be pure testability issues
> >> >> - Not confirmed to be rare condition cases
> >> >>
> >> >> It would be nice if someone familiar with these components can take a
> >> look
> >> >> into these issues.
> >> >>
> >> >> - https://issues.apache.org/jira/browse/FLINK-17159 (ES6)
> >> >> - https://issues.apache.org/jira/browse/FLINK-17912 (Kafka)
> >> >> - https://issues.apache.org/jira/browse/FLINK-17949 (Kafka)
> >> >> ⁃ https://issues.apache.org/jira/browse/FLINK-18444 (Kafka)
> >> >> - https://issues.apache.org/jira/browse/FLINK-18634 (Kafka)
> >> >> - https://issues.apache.org/jira/browse/FLINK-18648 (Kafka)
> >> >> - https://issues.apache.org/jira/browse/FLINK-18807 (Kafka)
> >> >> - https://issues.apache.org/jira/browse/FLINK-19369 (BlobClientTest)
> >> >> - https://issues.apache.org/jira/browse/FLINK-19436 (TPCDS)
> >> >> - https://issues.apache.org/jira/browse/FLINK-19690 (Format/Parquet)
> >> >> - https://issues.apache.org/jira/browse/FLINK-19775
> >> >> (SystemProcessingTimeServiceTest)
> >> >>
> >> >> Thank you~
> >> >>
> >> >> Xintong Song
> >> >>
> >> >>
> >> >>
> >> >> On Thu, Oct 29, 2020 at 10:21 AM Jingsong Li <jingsonglee0@gmail.com
> >
> >> >> wrote:
> >> >>
> >> >> > +1 to backport the FLIP-27 adjustments to 1.11.x.
> >> >> >
> >> >> > If possible, that would be great. Many people are looking forward
> to
> >> the
> >> >> > FLIP-27 interface, but they don't want to take the risk to upgrade
> to
> >> >> 1.12
> >> >> > (And wait 1.12). After all, 1.11 is a relatively stable version.
> >> >> >
> >> >> > Best,
> >> >> > Jingsong
> >> >> >
> >> >> > On Thu, Oct 29, 2020 at 1:24 AM Stephan Ewen <se...@apache.org>
> >> wrote:
> >> >> >
> >> >> > > Thanks for starting this.
> >> >> > >
> >> >> > > +1 form my side to backport the FLIP-27 adjustments to 1.11.x.
> >> >> > >
> >> >> > > There were quite a few changes, and I think we need to
> cherry-pick
> >> >> them
> >> >> > all
> >> >> > > to not get some inconsistent mix of changes and many merge
> >> conflicts.
> >> >> > > I made a list below of what we need to add to "release-1.11".
> >> >> > >
> >> >> > > *#### Core Source API Changes to backport (in REVERSE order)*
> >> >> > >
> >> >> > >   (Use: "git log
> >> >> > > flink-core/src/main/java/org/apache/flink/api/connector/source")
> >> >> > >
> >> >> > > commit 162c072e9265a7b6dd9d6f5459eb7974091c4c4e
> >> >> > > [FLINK-19492][core] Consolidate Source Events between Source API
> >> and
> >> >> > Split
> >> >> > > Reader API
> >> >> > >
> >> >> > > commit 428fbbad85379f71a9fbf1a1ab3e3dcfa9006436
> >> >> > > [FLINK-19205][core] Add access to configuration and hostname in
> the
> >> >> > > SourceReaderContext
> >> >> > >
> >> >> > > commit 723e1790744ddc78d19c7c978442af1383f38d33
> >> >> > > [hotfix][core] Add to Source Enumerator convenience methods to
> >> assign
> >> >> > > single split
> >> >> > >
> >> >> > > *#### Connector Base Changes to Backport (in REVERSE order)*
> >> >> > >
> >> >> > >   (Use: "git log flink-connectors/flink-connector-base")
> >> >> > >
> >> >> > > commit 401f56fe9d6b0271260edf9787cdcbfe4d03874d
> >> >> > > [FLINK-19427][FLINK-19489][tests] Fix test conditions for
> >> >> > > 'SplitFetcherTest.testNotifiesWhenGoingIdleConcurrent()'
> >> >> > >
> >> >> > > commit 68c5c2ff779d82a1ff81ffaf60d8a1b283797db1
> >> >> > > [FLINK-19448][connector base] Explicitly check for un-expected
> >> >> condition
> >> >> > > that would leave an inconsistent state
> >> >> > >
> >> >> > > commit 162c072e9265a7b6dd9d6f5459eb7974091c4c4e
> >> >> > > [FLINK-19492][core] Consolidate Source Events between Source API
> >> and
> >> >> > Split
> >> >> > > Reader API
> >> >> > >
> >> >> > > commit c1ca7a4c7c21ec8868c14cdddd43c559625b794c
> >> >> > > [refactor][tests] Move some source test utils from
> >> >> flink-connector-base
> >> >> > to
> >> >> > > flink-core
> >> >> > >
> >> >> > > commit ee5c4c211c35c70d28252363bbc8400453609977
> >> >> > > [FLINK-19251][connectors] Avoid confusing queue handling in
> >> >> > > "SplitReader.handleSplitsChanges()"
> >> >> > >
> >> >> > > commit 5abef56b2bf85bcac786f6b16b6899b6cced7176
> >> >> > > [FLINK-19250][connectors] Fix error propagation in connector base
> >> >> > > (SplitFetcherManager).
> >> >> > >
> >> >> > > commit 8fcca837c55a9216595ee4c03038b52747098dbb
> >> >> > > [hotfix][connectors] Improve JavaDocs for
> >> SingleThreadFetcherManager
> >> >> > >
> >> >> > > commit 4700bb5dde3303cbe98882f6beb7379425717b01
> >> >> > > [FLINK-19225][connectors] Various small improvements to
> >> >> SourceReaderBase
> >> >> > > (part 2)
> >> >> > >
> >> >> > > commit 12261c6b7ed6478a9b9f6a69cb58246b83cab9b7
> >> >> > > [FLINK-17393][connectors] (follow-up) Wakeup the SplitFetchers
> more
> >> >> > > elegantly.
> >> >> > >
> >> >> > > commit c60aaff0249bfd6b5871b7f82e03efc487a54d6b
> >> >> > > [hotfix][tests] Extend test coverage for
> >> FutureCompletingBlockingQueue
> >> >> > >
> >> >> > > commit cef8a587d7fd2fe64cc644da5ed095d82e46f631
> >> >> > > [FLINK-19245][connectors] Set default capacity for
> >> >> > > FutureCompletingBlockingQueue.
> >> >> > >
> >> >> > > commit 4ea95782b4c6a2538153d4d16ad3f4839c7de0fb
> >> >> > > [FLINK-19223][connectors] Simplify Availability Future Model in
> >> Base
> >> >> > > Connector
> >> >> > >
> >> >> > > commit 511857049ba30c8ff0ee56da551fa4a479dc583e
> >> >> > > [FLINK-18128][connectors] Ensure idle split fetchers lead to
> >> >> availability
> >> >> > > notifications.
> >> >> > >
> >> >> > > commit a8206467af0830dcb89623ea068b5ca3b3450c92
> >> >> > > [refactor][core] Eagerly initialize the FetchTask to support
> proper
> >> >> unit
> >> >> > > testing
> >> >> > >
> >> >> > > commit 3b2f54bcb437f98e6137c904045cc51072b5c06b
> >> >> > > [hotfix][tests] Move constants in SplitFetcherTest relevant to
> only
> >> >> one
> >> >> > > test into test method
> >> >> > >
> >> >> > > commit d7625760a75a508bf05bcddc380bb4d62ee1743e
> >> >> > > [FLINK-19225][connectors] Various small improvements to
> >> >> SourceReaderBase
> >> >> > >
> >> >> > > commit a5b0d3297748c1be47ad579a88f24df2255a8df1
> >> >> > > [FLINK-17393][connectors] Wakeup the SplitFetchers more
> elegantly.
> >> >> > >
> >> >> > > commit f42a3ebc3e81a034b7221a803c153636fef34903
> >> >> > > [FLINK-18680][connectors] Make connector base RecordsWithSplitIds
> >> more
> >> >> > > lightweight.
> >> >> > >
> >> >> > > commit e3d273de822b085183d09b275a445879ff94b350
> >> >> > > [FLINK-19162][connectors] Add 'recycle()' to the
> >> RecordsWithSplitIds
> >> >> to
> >> >> > > support reuse of heavy objects.
> >> >> > >
> >> >> > > commit 8ebc464c2520453a70001cd712abc8dee6ee89e0
> >> >> > > [hotfix][testing] Add a set of parameterizable testing mocks for
> >> the
> >> >> > Split
> >> >> > > Reader API
> >> >> > >
> >> >> > > commit 930a07438be1185388d7150640f294dfe2a5d378
> >> >> > > [hotfix][connectors] Add RequestSplitEvent to
> >> 'flink-connector-base'
> >> >> > >
> >> >> > > commit e945ce8a933bc378844782f784ca473c767ca159
> >> >> > > [refactor][connectors] Backport of the connector-base exception
> >> >> handling
> >> >> > > from the Kafka Connector Pull Request
> >> >> > >
> >> >> > >
> >> >> > > On Wed, Oct 28, 2020 at 5:49 PM Steven Wu <st...@gmail.com>
> >> >> wrote:
> >> >> > >
> >> >> > > > I would love to see this FLIP-27 source interface improvement
> [1]
> >> >> made
> >> >> > to
> >> >> > > > 1.11.3.
> >> >> > > >
> >> >> > > > [1] https://issues.apache.org/jira/browse/FLINK-19698
> >> >> > > >
> >> >> > > > On Wed, Oct 28, 2020 at 12:32 AM Tzu-Li (Gordon) Tai <
> >> >> > > tzulitai@apache.org>
> >> >> > > > wrote:
> >> >> > > >
> >> >> > > > > Thanks for the replies so far!
> >> >> > > > >
> >> >> > > > > Just to provide a brief update on the status of blockers for
> >> >> 1.11.3
> >> >> > so
> >> >> > > > far:
> >> >> > > > >
> >> >> > > > >
> >> >> > > > > *PR opened, pending reviewer*- [FLINK-19717]
> >> >> > SourceReaderBase.pollNext
> >> >> > > > may
> >> >> > > > > return END_OF_INPUT if SplitReader.fetch throws (
> >> >> > > > > https://github.com/apache/flink/pull/13776)
> >> >> > > > >
> >> >> > > > > *PR opened, reviewed + close to being merged*
> >> >> > > > > - [FLINK-19741] Timer service should skip restoring from raw
> >> keyed
> >> >> > > stream
> >> >> > > > > if it isn't the writer (
> >> >> https://github.com/apache/flink/pull/13761)
> >> >> > > > > - [FLINK-19748] Raw keyed stream key group iterator should be
> >> >> > skipping
> >> >> > > > > unwritten key groups (
> >> https://github.com/apache/flink/pull/13772)
> >> >> > > > >
> >> >> > > > > *Merged*
> >> >> > > > > - [FLINK-19154] Application mode deletes HA data in case of
> >> >> suspended
> >> >> > > > > ZooKeeper connection
> >> >> > > > > - [FLINK-19569] Upgrade ICU4J to 67.1+
> >> >> > > > >
> >> >> > > > > Right now as it seems, progress is mainly blocked on a
> reviewer
> >> >> for
> >> >> > > > > FLINK-19717.
> >> >> > > > > Meanwhile, Xintong is keeping an eye on test instabilities
> [1]
> >> to
> >> >> see
> >> >> > > if
> >> >> > > > > there are any fixes that should be applied to `release-1.11`.
> >> >> > > > >
> >> >> > > > > This is also a reminder, that if there are other blockers
> that
> >> we
> >> >> > need
> >> >> > > to
> >> >> > > > > be aware of, or a need to re-establish estimated time for
> >> getting
> >> >> > fixes
> >> >> > > > in
> >> >> > > > > and delay the RC for 1.11.3, please do let us know!
> >> >> > > > >
> >> >> > > > > Cheers,
> >> >> > > > > Gordon
> >> >> > > > >
> >> >> > > > > [1]
> >> >> > > > >
> >> >> > > > >
> >> >> > > >
> >> >> > >
> >> >> >
> >> >>
> >>
> https://issues.apache.org/jira/issues/?jql=project%20%3D%20FLINK%20AND%20status%20in%20(Open%2C%20%22In%20Progress%22%2C%20Reopened)%20AND%20affectedVersion%20in%20(1.11.0%2C%201.11.1%2C%201.11.2)%20AND%20labels%20%3D%20test-stability
> >> >> > > > >
> >> >> > > > > On Mon, Oct 26, 2020 at 9:43 PM Kostas Kloudas <
> >> >> kkloudas@apache.org>
> >> >> > > > > wrote:
> >> >> > > > >
> >> >> > > > > > +1 for releasing Flink 1.11.3 as it contains a number of
> >> >> important
> >> >> > > > > > fixes and thanks Gordon and Xintong for volunteering.
> >> >> > > > > >
> >> >> > > > > > Cheers,
> >> >> > > > > > Kostas
> >> >> > > > > >
> >> >> > > > > > On Mon, Oct 26, 2020 at 4:37 AM Yu Li <ca...@gmail.com>
> >> wrote:
> >> >> > > > > > >
> >> >> > > > > > > +1 for releasing Flink 1.11.3, and thanks Gordon and
> >> Xintong
> >> >> for
> >> >> > > > > > > volunteering as our release managers.
> >> >> > > > > > >
> >> >> > > > > > > Best Regards,
> >> >> > > > > > > Yu
> >> >> > > > > > >
> >> >> > > > > > >
> >> >> > > > > > > On Mon, 26 Oct 2020 at 09:45, Xintong Song <
> >> >> > tonysong820@gmail.com>
> >> >> > > > > > wrote:
> >> >> > > > > > >
> >> >> > > > > > > > Thanks Gordan for starting this discussion.
> >> >> > > > > > > > My pleasure to help with the release process.
> >> >> > > > > > > >
> >> >> > > > > > > > Thank you~
> >> >> > > > > > > >
> >> >> > > > > > > > Xintong Song
> >> >> > > > > > > >
> >> >> > > > > > > >
> >> >> > > > > > > >
> >> >> > > > > > > > On Fri, Oct 23, 2020 at 11:29 PM Till Rohrmann <
> >> >> > > > trohrmann@apache.org
> >> >> > > > > >
> >> >> > > > > > > > wrote:
> >> >> > > > > > > >
> >> >> > > > > > > > > Thanks for starting this discussion Gordon. There are
> >> over
> >> >> > 100
> >> >> > > > > issues
> >> >> > > > > > > > > which are fixed for 1.11.3. Hence +1 for a soonish
> >> 1.11.3
> >> >> > > > release.
> >> >> > > > > > Thanks
> >> >> > > > > > > > > for volunteering as our release managers Gordon and
> >> >> Xintong!
> >> >> > > > > > > > >
> >> >> > > > > > > > > Cheers,
> >> >> > > > > > > > > Till
> >> >> > > > > > > > >
> >> >> > > > > > > > > On Fri, Oct 23, 2020 at 5:02 PM Tzu-Li (Gordon) Tai <
> >> >> > > > > > tzulitai@apache.org
> >> >> > > > > > > > >
> >> >> > > > > > > > > wrote:
> >> >> > > > > > > > >
> >> >> > > > > > > > >> Hi,
> >> >> > > > > > > > >>
> >> >> > > > > > > > >> Xintong and I would like to start a discussion for
> >> >> releasing
> >> >> > > > Flink
> >> >> > > > > > > > 1.11.3
> >> >> > > > > > > > >> soon.
> >> >> > > > > > > > >>
> >> >> > > > > > > > >> It seems like we already have a few pressing issues
> >> that
> >> >> > needs
> >> >> > > > to
> >> >> > > > > be
> >> >> > > > > > > > >> included in a new hotfix release:
> >> >> > > > > > > > >>
> >> >> > > > > > > > >>    - Heap-based timers’ restore behaviour is
> causing a
> >> >> > > critical
> >> >> > > > > > recovery
> >> >> > > > > > > > >>    issue for StateFun [1] [2] [3].
> >> >> > > > > > > > >>    - There are several robustness issues for the
> >> FLIP-27
> >> >> new
> >> >> > > > > source
> >> >> > > > > > API,
> >> >> > > > > > > > >>    such as [4]. We already have some users using the
> >> >> FLIP-27
> >> >> > > API
> >> >> > > > > > with
> >> >> > > > > > > > >> 1.11.x,
> >> >> > > > > > > > >>    so it would be important to get those fixes in
> for
> >> >> 1.11.x
> >> >> > > as
> >> >> > > > > > well.
> >> >> > > > > > > > >>
> >> >> > > > > > > > >> Apart from the issues that are already marked as
> >> blocker
> >> >> for
> >> >> > > > > 1.11.3
> >> >> > > > > > in
> >> >> > > > > > > > our
> >> >> > > > > > > > >> JIRA [5], please let us know in this thread if there
> >> is
> >> >> > > already
> >> >> > > > > > ongoing
> >> >> > > > > > > > >> work for other important fixes that we should try to
> >> >> > include.
> >> >> > > > > > > > >>
> >> >> > > > > > > > >> Xintong and I would like to volunteer for managing
> >> this
> >> >> > > release,
> >> >> > > > > and
> >> >> > > > > > > > will
> >> >> > > > > > > > >> try to communicate the priority of pending blockers
> >> over
> >> >> the
> >> >> > > > next
> >> >> > > > > > few
> >> >> > > > > > > > >> days.
> >> >> > > > > > > > >> Since the aforementioned issues are quite critical,
> >> we’d
> >> >> > like
> >> >> > > to
> >> >> > > > > aim
> >> >> > > > > > > > >> for a *feature
> >> >> > > > > > > > >> freeze by the end of next week (Oct. 30th)* and
> start
> >> the
> >> >> > > > release
> >> >> > > > > > voting
> >> >> > > > > > > > >> process the week after.
> >> >> > > > > > > > >> If that is too short of a notice and you might need
> >> more
> >> >> > time,
> >> >> > > > > > please
> >> >> > > > > > > > let
> >> >> > > > > > > > >> us know!
> >> >> > > > > > > > >>
> >> >> > > > > > > > >> Cheers,
> >> >> > > > > > > > >> Gordon
> >> >> > > > > > > > >>
> >> >> > > > > > > > >> [1]
> https://issues.apache.org/jira/browse/FLINK-19692
> >> >> > > > > > > > >> [2]
> https://issues.apache.org/jira/browse/FLINK-19741
> >> >> > > > > > > > >> [3]
> https://issues.apache.org/jira/browse/FLINK-19748
> >> >> > > > > > > > >> [4]
> https://issues.apache.org/jira/browse/FLINK-19717
> >> >> > > > > > > > >> [5]
> >> >> > > > > > > > >>
> >> >> > > > > > > > >>
> >> >> > > > > > > >
> >> >> > > > > >
> >> >> > > > >
> >> >> > > >
> >> >> > >
> >> >> >
> >> >>
> >>
> https://issues.apache.org/jira/issues/?jql=project%20%3D%20FLINK%20AND%20priority%20%3D%20Blocker%20AND%20fixVersion%20%3D%201.11.3
> >> >> > > > > > > > >>
> >> >> > > > > > > > >
> >> >> > > > > > > >
> >> >> > > > > >
> >> >> > > > >
> >> >> > > >
> >> >> > >
> >> >> >
> >> >> >
> >> >> > --
> >> >> > Best, Jingsong Lee
> >> >> >
> >> >>
> >> >
> >>
> >
>

Re: [DISCUSS] Releasing Apache Flink 1.11.3

Posted by Steven Wu <st...@gmail.com>.
Basically, it would be great to get the latest code in the
flink-connector-files (FLIP-27).

On Sat, Oct 31, 2020 at 9:57 AM Steven Wu <st...@gmail.com> wrote:

> Stephan, it will be great if we can also backport the DataStreamUtils
> related commits that help with collecting output from unbounded streams.
> e.g.
>
>
> https://github.com/apache/flink/commit/09a7a66b7313fea64817fe960a8da1265b428efc
>
> I tried to copy and paste the code to unblock myself. but it quickly got
> into the rabbit hole of more and more code.
>
> On Fri, Oct 30, 2020 at 11:02 AM Stephan Ewen <se...@apache.org> wrote:
>
>> I have started with backporting the source API changes. Some minor
>> conflicts to solve, will need a bit more to finish this.
>>
>> On Fri, Oct 30, 2020 at 7:25 AM Tzu-Li (Gordon) Tai <tz...@apache.org>
>> wrote:
>>
>> > @Stephan Ewen <se...@apache.org>
>> > Are there already plans or ongoing efforts for backporting the list of
>> > FLIP-27 changes that you posted?
>> >
>> > On Thu, Oct 29, 2020 at 7:08 PM Xintong Song <to...@gmail.com>
>> > wrote:
>> >
>> >> Hi folks,
>> >>
>> >> Just to provide some updates concerning the status on the
>> >> test instabilities.
>> >>
>> >> Currently, we have 30 unresolved tickets labeled with `Affects Version`
>> >> 1.11.x.
>> >>
>> >>
>> https://issues.apache.org/jira/browse/FLINK-19775?filter=12348580&jql=project%20%3D%20FLINK%20AND%20resolution%20%3D%20Unresolved%20AND%20affectedVersion%20in%20(1.11.0%2C%201.11.1%2C%201.11.2%2C%201.11.3)%20AND%20labels%20%3D%20test-stability%20ORDER%20BY%20created%20DESC
>> >>
>> >> Among the 30 tickets, 11 of them are:
>> >> - Have occured in the recent 3 months
>> >> - Not confirmed to be pure testability issues
>> >> - Not confirmed to be rare condition cases
>> >>
>> >> It would be nice if someone familiar with these components can take a
>> look
>> >> into these issues.
>> >>
>> >> - https://issues.apache.org/jira/browse/FLINK-17159 (ES6)
>> >> - https://issues.apache.org/jira/browse/FLINK-17912 (Kafka)
>> >> - https://issues.apache.org/jira/browse/FLINK-17949 (Kafka)
>> >> ⁃ https://issues.apache.org/jira/browse/FLINK-18444 (Kafka)
>> >> - https://issues.apache.org/jira/browse/FLINK-18634 (Kafka)
>> >> - https://issues.apache.org/jira/browse/FLINK-18648 (Kafka)
>> >> - https://issues.apache.org/jira/browse/FLINK-18807 (Kafka)
>> >> - https://issues.apache.org/jira/browse/FLINK-19369 (BlobClientTest)
>> >> - https://issues.apache.org/jira/browse/FLINK-19436 (TPCDS)
>> >> - https://issues.apache.org/jira/browse/FLINK-19690 (Format/Parquet)
>> >> - https://issues.apache.org/jira/browse/FLINK-19775
>> >> (SystemProcessingTimeServiceTest)
>> >>
>> >> Thank you~
>> >>
>> >> Xintong Song
>> >>
>> >>
>> >>
>> >> On Thu, Oct 29, 2020 at 10:21 AM Jingsong Li <ji...@gmail.com>
>> >> wrote:
>> >>
>> >> > +1 to backport the FLIP-27 adjustments to 1.11.x.
>> >> >
>> >> > If possible, that would be great. Many people are looking forward to
>> the
>> >> > FLIP-27 interface, but they don't want to take the risk to upgrade to
>> >> 1.12
>> >> > (And wait 1.12). After all, 1.11 is a relatively stable version.
>> >> >
>> >> > Best,
>> >> > Jingsong
>> >> >
>> >> > On Thu, Oct 29, 2020 at 1:24 AM Stephan Ewen <se...@apache.org>
>> wrote:
>> >> >
>> >> > > Thanks for starting this.
>> >> > >
>> >> > > +1 form my side to backport the FLIP-27 adjustments to 1.11.x.
>> >> > >
>> >> > > There were quite a few changes, and I think we need to cherry-pick
>> >> them
>> >> > all
>> >> > > to not get some inconsistent mix of changes and many merge
>> conflicts.
>> >> > > I made a list below of what we need to add to "release-1.11".
>> >> > >
>> >> > > *#### Core Source API Changes to backport (in REVERSE order)*
>> >> > >
>> >> > >   (Use: "git log
>> >> > > flink-core/src/main/java/org/apache/flink/api/connector/source")
>> >> > >
>> >> > > commit 162c072e9265a7b6dd9d6f5459eb7974091c4c4e
>> >> > > [FLINK-19492][core] Consolidate Source Events between Source API
>> and
>> >> > Split
>> >> > > Reader API
>> >> > >
>> >> > > commit 428fbbad85379f71a9fbf1a1ab3e3dcfa9006436
>> >> > > [FLINK-19205][core] Add access to configuration and hostname in the
>> >> > > SourceReaderContext
>> >> > >
>> >> > > commit 723e1790744ddc78d19c7c978442af1383f38d33
>> >> > > [hotfix][core] Add to Source Enumerator convenience methods to
>> assign
>> >> > > single split
>> >> > >
>> >> > > *#### Connector Base Changes to Backport (in REVERSE order)*
>> >> > >
>> >> > >   (Use: "git log flink-connectors/flink-connector-base")
>> >> > >
>> >> > > commit 401f56fe9d6b0271260edf9787cdcbfe4d03874d
>> >> > > [FLINK-19427][FLINK-19489][tests] Fix test conditions for
>> >> > > 'SplitFetcherTest.testNotifiesWhenGoingIdleConcurrent()'
>> >> > >
>> >> > > commit 68c5c2ff779d82a1ff81ffaf60d8a1b283797db1
>> >> > > [FLINK-19448][connector base] Explicitly check for un-expected
>> >> condition
>> >> > > that would leave an inconsistent state
>> >> > >
>> >> > > commit 162c072e9265a7b6dd9d6f5459eb7974091c4c4e
>> >> > > [FLINK-19492][core] Consolidate Source Events between Source API
>> and
>> >> > Split
>> >> > > Reader API
>> >> > >
>> >> > > commit c1ca7a4c7c21ec8868c14cdddd43c559625b794c
>> >> > > [refactor][tests] Move some source test utils from
>> >> flink-connector-base
>> >> > to
>> >> > > flink-core
>> >> > >
>> >> > > commit ee5c4c211c35c70d28252363bbc8400453609977
>> >> > > [FLINK-19251][connectors] Avoid confusing queue handling in
>> >> > > "SplitReader.handleSplitsChanges()"
>> >> > >
>> >> > > commit 5abef56b2bf85bcac786f6b16b6899b6cced7176
>> >> > > [FLINK-19250][connectors] Fix error propagation in connector base
>> >> > > (SplitFetcherManager).
>> >> > >
>> >> > > commit 8fcca837c55a9216595ee4c03038b52747098dbb
>> >> > > [hotfix][connectors] Improve JavaDocs for
>> SingleThreadFetcherManager
>> >> > >
>> >> > > commit 4700bb5dde3303cbe98882f6beb7379425717b01
>> >> > > [FLINK-19225][connectors] Various small improvements to
>> >> SourceReaderBase
>> >> > > (part 2)
>> >> > >
>> >> > > commit 12261c6b7ed6478a9b9f6a69cb58246b83cab9b7
>> >> > > [FLINK-17393][connectors] (follow-up) Wakeup the SplitFetchers more
>> >> > > elegantly.
>> >> > >
>> >> > > commit c60aaff0249bfd6b5871b7f82e03efc487a54d6b
>> >> > > [hotfix][tests] Extend test coverage for
>> FutureCompletingBlockingQueue
>> >> > >
>> >> > > commit cef8a587d7fd2fe64cc644da5ed095d82e46f631
>> >> > > [FLINK-19245][connectors] Set default capacity for
>> >> > > FutureCompletingBlockingQueue.
>> >> > >
>> >> > > commit 4ea95782b4c6a2538153d4d16ad3f4839c7de0fb
>> >> > > [FLINK-19223][connectors] Simplify Availability Future Model in
>> Base
>> >> > > Connector
>> >> > >
>> >> > > commit 511857049ba30c8ff0ee56da551fa4a479dc583e
>> >> > > [FLINK-18128][connectors] Ensure idle split fetchers lead to
>> >> availability
>> >> > > notifications.
>> >> > >
>> >> > > commit a8206467af0830dcb89623ea068b5ca3b3450c92
>> >> > > [refactor][core] Eagerly initialize the FetchTask to support proper
>> >> unit
>> >> > > testing
>> >> > >
>> >> > > commit 3b2f54bcb437f98e6137c904045cc51072b5c06b
>> >> > > [hotfix][tests] Move constants in SplitFetcherTest relevant to only
>> >> one
>> >> > > test into test method
>> >> > >
>> >> > > commit d7625760a75a508bf05bcddc380bb4d62ee1743e
>> >> > > [FLINK-19225][connectors] Various small improvements to
>> >> SourceReaderBase
>> >> > >
>> >> > > commit a5b0d3297748c1be47ad579a88f24df2255a8df1
>> >> > > [FLINK-17393][connectors] Wakeup the SplitFetchers more elegantly.
>> >> > >
>> >> > > commit f42a3ebc3e81a034b7221a803c153636fef34903
>> >> > > [FLINK-18680][connectors] Make connector base RecordsWithSplitIds
>> more
>> >> > > lightweight.
>> >> > >
>> >> > > commit e3d273de822b085183d09b275a445879ff94b350
>> >> > > [FLINK-19162][connectors] Add 'recycle()' to the
>> RecordsWithSplitIds
>> >> to
>> >> > > support reuse of heavy objects.
>> >> > >
>> >> > > commit 8ebc464c2520453a70001cd712abc8dee6ee89e0
>> >> > > [hotfix][testing] Add a set of parameterizable testing mocks for
>> the
>> >> > Split
>> >> > > Reader API
>> >> > >
>> >> > > commit 930a07438be1185388d7150640f294dfe2a5d378
>> >> > > [hotfix][connectors] Add RequestSplitEvent to
>> 'flink-connector-base'
>> >> > >
>> >> > > commit e945ce8a933bc378844782f784ca473c767ca159
>> >> > > [refactor][connectors] Backport of the connector-base exception
>> >> handling
>> >> > > from the Kafka Connector Pull Request
>> >> > >
>> >> > >
>> >> > > On Wed, Oct 28, 2020 at 5:49 PM Steven Wu <st...@gmail.com>
>> >> wrote:
>> >> > >
>> >> > > > I would love to see this FLIP-27 source interface improvement [1]
>> >> made
>> >> > to
>> >> > > > 1.11.3.
>> >> > > >
>> >> > > > [1] https://issues.apache.org/jira/browse/FLINK-19698
>> >> > > >
>> >> > > > On Wed, Oct 28, 2020 at 12:32 AM Tzu-Li (Gordon) Tai <
>> >> > > tzulitai@apache.org>
>> >> > > > wrote:
>> >> > > >
>> >> > > > > Thanks for the replies so far!
>> >> > > > >
>> >> > > > > Just to provide a brief update on the status of blockers for
>> >> 1.11.3
>> >> > so
>> >> > > > far:
>> >> > > > >
>> >> > > > >
>> >> > > > > *PR opened, pending reviewer*- [FLINK-19717]
>> >> > SourceReaderBase.pollNext
>> >> > > > may
>> >> > > > > return END_OF_INPUT if SplitReader.fetch throws (
>> >> > > > > https://github.com/apache/flink/pull/13776)
>> >> > > > >
>> >> > > > > *PR opened, reviewed + close to being merged*
>> >> > > > > - [FLINK-19741] Timer service should skip restoring from raw
>> keyed
>> >> > > stream
>> >> > > > > if it isn't the writer (
>> >> https://github.com/apache/flink/pull/13761)
>> >> > > > > - [FLINK-19748] Raw keyed stream key group iterator should be
>> >> > skipping
>> >> > > > > unwritten key groups (
>> https://github.com/apache/flink/pull/13772)
>> >> > > > >
>> >> > > > > *Merged*
>> >> > > > > - [FLINK-19154] Application mode deletes HA data in case of
>> >> suspended
>> >> > > > > ZooKeeper connection
>> >> > > > > - [FLINK-19569] Upgrade ICU4J to 67.1+
>> >> > > > >
>> >> > > > > Right now as it seems, progress is mainly blocked on a reviewer
>> >> for
>> >> > > > > FLINK-19717.
>> >> > > > > Meanwhile, Xintong is keeping an eye on test instabilities [1]
>> to
>> >> see
>> >> > > if
>> >> > > > > there are any fixes that should be applied to `release-1.11`.
>> >> > > > >
>> >> > > > > This is also a reminder, that if there are other blockers that
>> we
>> >> > need
>> >> > > to
>> >> > > > > be aware of, or a need to re-establish estimated time for
>> getting
>> >> > fixes
>> >> > > > in
>> >> > > > > and delay the RC for 1.11.3, please do let us know!
>> >> > > > >
>> >> > > > > Cheers,
>> >> > > > > Gordon
>> >> > > > >
>> >> > > > > [1]
>> >> > > > >
>> >> > > > >
>> >> > > >
>> >> > >
>> >> >
>> >>
>> https://issues.apache.org/jira/issues/?jql=project%20%3D%20FLINK%20AND%20status%20in%20(Open%2C%20%22In%20Progress%22%2C%20Reopened)%20AND%20affectedVersion%20in%20(1.11.0%2C%201.11.1%2C%201.11.2)%20AND%20labels%20%3D%20test-stability
>> >> > > > >
>> >> > > > > On Mon, Oct 26, 2020 at 9:43 PM Kostas Kloudas <
>> >> kkloudas@apache.org>
>> >> > > > > wrote:
>> >> > > > >
>> >> > > > > > +1 for releasing Flink 1.11.3 as it contains a number of
>> >> important
>> >> > > > > > fixes and thanks Gordon and Xintong for volunteering.
>> >> > > > > >
>> >> > > > > > Cheers,
>> >> > > > > > Kostas
>> >> > > > > >
>> >> > > > > > On Mon, Oct 26, 2020 at 4:37 AM Yu Li <ca...@gmail.com>
>> wrote:
>> >> > > > > > >
>> >> > > > > > > +1 for releasing Flink 1.11.3, and thanks Gordon and
>> Xintong
>> >> for
>> >> > > > > > > volunteering as our release managers.
>> >> > > > > > >
>> >> > > > > > > Best Regards,
>> >> > > > > > > Yu
>> >> > > > > > >
>> >> > > > > > >
>> >> > > > > > > On Mon, 26 Oct 2020 at 09:45, Xintong Song <
>> >> > tonysong820@gmail.com>
>> >> > > > > > wrote:
>> >> > > > > > >
>> >> > > > > > > > Thanks Gordan for starting this discussion.
>> >> > > > > > > > My pleasure to help with the release process.
>> >> > > > > > > >
>> >> > > > > > > > Thank you~
>> >> > > > > > > >
>> >> > > > > > > > Xintong Song
>> >> > > > > > > >
>> >> > > > > > > >
>> >> > > > > > > >
>> >> > > > > > > > On Fri, Oct 23, 2020 at 11:29 PM Till Rohrmann <
>> >> > > > trohrmann@apache.org
>> >> > > > > >
>> >> > > > > > > > wrote:
>> >> > > > > > > >
>> >> > > > > > > > > Thanks for starting this discussion Gordon. There are
>> over
>> >> > 100
>> >> > > > > issues
>> >> > > > > > > > > which are fixed for 1.11.3. Hence +1 for a soonish
>> 1.11.3
>> >> > > > release.
>> >> > > > > > Thanks
>> >> > > > > > > > > for volunteering as our release managers Gordon and
>> >> Xintong!
>> >> > > > > > > > >
>> >> > > > > > > > > Cheers,
>> >> > > > > > > > > Till
>> >> > > > > > > > >
>> >> > > > > > > > > On Fri, Oct 23, 2020 at 5:02 PM Tzu-Li (Gordon) Tai <
>> >> > > > > > tzulitai@apache.org
>> >> > > > > > > > >
>> >> > > > > > > > > wrote:
>> >> > > > > > > > >
>> >> > > > > > > > >> Hi,
>> >> > > > > > > > >>
>> >> > > > > > > > >> Xintong and I would like to start a discussion for
>> >> releasing
>> >> > > > Flink
>> >> > > > > > > > 1.11.3
>> >> > > > > > > > >> soon.
>> >> > > > > > > > >>
>> >> > > > > > > > >> It seems like we already have a few pressing issues
>> that
>> >> > needs
>> >> > > > to
>> >> > > > > be
>> >> > > > > > > > >> included in a new hotfix release:
>> >> > > > > > > > >>
>> >> > > > > > > > >>    - Heap-based timers’ restore behaviour is causing a
>> >> > > critical
>> >> > > > > > recovery
>> >> > > > > > > > >>    issue for StateFun [1] [2] [3].
>> >> > > > > > > > >>    - There are several robustness issues for the
>> FLIP-27
>> >> new
>> >> > > > > source
>> >> > > > > > API,
>> >> > > > > > > > >>    such as [4]. We already have some users using the
>> >> FLIP-27
>> >> > > API
>> >> > > > > > with
>> >> > > > > > > > >> 1.11.x,
>> >> > > > > > > > >>    so it would be important to get those fixes in for
>> >> 1.11.x
>> >> > > as
>> >> > > > > > well.
>> >> > > > > > > > >>
>> >> > > > > > > > >> Apart from the issues that are already marked as
>> blocker
>> >> for
>> >> > > > > 1.11.3
>> >> > > > > > in
>> >> > > > > > > > our
>> >> > > > > > > > >> JIRA [5], please let us know in this thread if there
>> is
>> >> > > already
>> >> > > > > > ongoing
>> >> > > > > > > > >> work for other important fixes that we should try to
>> >> > include.
>> >> > > > > > > > >>
>> >> > > > > > > > >> Xintong and I would like to volunteer for managing
>> this
>> >> > > release,
>> >> > > > > and
>> >> > > > > > > > will
>> >> > > > > > > > >> try to communicate the priority of pending blockers
>> over
>> >> the
>> >> > > > next
>> >> > > > > > few
>> >> > > > > > > > >> days.
>> >> > > > > > > > >> Since the aforementioned issues are quite critical,
>> we’d
>> >> > like
>> >> > > to
>> >> > > > > aim
>> >> > > > > > > > >> for a *feature
>> >> > > > > > > > >> freeze by the end of next week (Oct. 30th)* and start
>> the
>> >> > > > release
>> >> > > > > > voting
>> >> > > > > > > > >> process the week after.
>> >> > > > > > > > >> If that is too short of a notice and you might need
>> more
>> >> > time,
>> >> > > > > > please
>> >> > > > > > > > let
>> >> > > > > > > > >> us know!
>> >> > > > > > > > >>
>> >> > > > > > > > >> Cheers,
>> >> > > > > > > > >> Gordon
>> >> > > > > > > > >>
>> >> > > > > > > > >> [1] https://issues.apache.org/jira/browse/FLINK-19692
>> >> > > > > > > > >> [2] https://issues.apache.org/jira/browse/FLINK-19741
>> >> > > > > > > > >> [3] https://issues.apache.org/jira/browse/FLINK-19748
>> >> > > > > > > > >> [4] https://issues.apache.org/jira/browse/FLINK-19717
>> >> > > > > > > > >> [5]
>> >> > > > > > > > >>
>> >> > > > > > > > >>
>> >> > > > > > > >
>> >> > > > > >
>> >> > > > >
>> >> > > >
>> >> > >
>> >> >
>> >>
>> https://issues.apache.org/jira/issues/?jql=project%20%3D%20FLINK%20AND%20priority%20%3D%20Blocker%20AND%20fixVersion%20%3D%201.11.3
>> >> > > > > > > > >>
>> >> > > > > > > > >
>> >> > > > > > > >
>> >> > > > > >
>> >> > > > >
>> >> > > >
>> >> > >
>> >> >
>> >> >
>> >> > --
>> >> > Best, Jingsong Lee
>> >> >
>> >>
>> >
>>
>

Re: [DISCUSS] Releasing Apache Flink 1.11.3

Posted by Steven Wu <st...@gmail.com>.
Stephan, it will be great if we can also backport the DataStreamUtils
related commits that help with collecting output from unbounded streams.
e.g.

https://github.com/apache/flink/commit/09a7a66b7313fea64817fe960a8da1265b428efc

I tried to copy and paste the code to unblock myself. but it quickly got
into the rabbit hole of more and more code.

On Fri, Oct 30, 2020 at 11:02 AM Stephan Ewen <se...@apache.org> wrote:

> I have started with backporting the source API changes. Some minor
> conflicts to solve, will need a bit more to finish this.
>
> On Fri, Oct 30, 2020 at 7:25 AM Tzu-Li (Gordon) Tai <tz...@apache.org>
> wrote:
>
> > @Stephan Ewen <se...@apache.org>
> > Are there already plans or ongoing efforts for backporting the list of
> > FLIP-27 changes that you posted?
> >
> > On Thu, Oct 29, 2020 at 7:08 PM Xintong Song <to...@gmail.com>
> > wrote:
> >
> >> Hi folks,
> >>
> >> Just to provide some updates concerning the status on the
> >> test instabilities.
> >>
> >> Currently, we have 30 unresolved tickets labeled with `Affects Version`
> >> 1.11.x.
> >>
> >>
> https://issues.apache.org/jira/browse/FLINK-19775?filter=12348580&jql=project%20%3D%20FLINK%20AND%20resolution%20%3D%20Unresolved%20AND%20affectedVersion%20in%20(1.11.0%2C%201.11.1%2C%201.11.2%2C%201.11.3)%20AND%20labels%20%3D%20test-stability%20ORDER%20BY%20created%20DESC
> >>
> >> Among the 30 tickets, 11 of them are:
> >> - Have occured in the recent 3 months
> >> - Not confirmed to be pure testability issues
> >> - Not confirmed to be rare condition cases
> >>
> >> It would be nice if someone familiar with these components can take a
> look
> >> into these issues.
> >>
> >> - https://issues.apache.org/jira/browse/FLINK-17159 (ES6)
> >> - https://issues.apache.org/jira/browse/FLINK-17912 (Kafka)
> >> - https://issues.apache.org/jira/browse/FLINK-17949 (Kafka)
> >> ⁃ https://issues.apache.org/jira/browse/FLINK-18444 (Kafka)
> >> - https://issues.apache.org/jira/browse/FLINK-18634 (Kafka)
> >> - https://issues.apache.org/jira/browse/FLINK-18648 (Kafka)
> >> - https://issues.apache.org/jira/browse/FLINK-18807 (Kafka)
> >> - https://issues.apache.org/jira/browse/FLINK-19369 (BlobClientTest)
> >> - https://issues.apache.org/jira/browse/FLINK-19436 (TPCDS)
> >> - https://issues.apache.org/jira/browse/FLINK-19690 (Format/Parquet)
> >> - https://issues.apache.org/jira/browse/FLINK-19775
> >> (SystemProcessingTimeServiceTest)
> >>
> >> Thank you~
> >>
> >> Xintong Song
> >>
> >>
> >>
> >> On Thu, Oct 29, 2020 at 10:21 AM Jingsong Li <ji...@gmail.com>
> >> wrote:
> >>
> >> > +1 to backport the FLIP-27 adjustments to 1.11.x.
> >> >
> >> > If possible, that would be great. Many people are looking forward to
> the
> >> > FLIP-27 interface, but they don't want to take the risk to upgrade to
> >> 1.12
> >> > (And wait 1.12). After all, 1.11 is a relatively stable version.
> >> >
> >> > Best,
> >> > Jingsong
> >> >
> >> > On Thu, Oct 29, 2020 at 1:24 AM Stephan Ewen <se...@apache.org>
> wrote:
> >> >
> >> > > Thanks for starting this.
> >> > >
> >> > > +1 form my side to backport the FLIP-27 adjustments to 1.11.x.
> >> > >
> >> > > There were quite a few changes, and I think we need to cherry-pick
> >> them
> >> > all
> >> > > to not get some inconsistent mix of changes and many merge
> conflicts.
> >> > > I made a list below of what we need to add to "release-1.11".
> >> > >
> >> > > *#### Core Source API Changes to backport (in REVERSE order)*
> >> > >
> >> > >   (Use: "git log
> >> > > flink-core/src/main/java/org/apache/flink/api/connector/source")
> >> > >
> >> > > commit 162c072e9265a7b6dd9d6f5459eb7974091c4c4e
> >> > > [FLINK-19492][core] Consolidate Source Events between Source API and
> >> > Split
> >> > > Reader API
> >> > >
> >> > > commit 428fbbad85379f71a9fbf1a1ab3e3dcfa9006436
> >> > > [FLINK-19205][core] Add access to configuration and hostname in the
> >> > > SourceReaderContext
> >> > >
> >> > > commit 723e1790744ddc78d19c7c978442af1383f38d33
> >> > > [hotfix][core] Add to Source Enumerator convenience methods to
> assign
> >> > > single split
> >> > >
> >> > > *#### Connector Base Changes to Backport (in REVERSE order)*
> >> > >
> >> > >   (Use: "git log flink-connectors/flink-connector-base")
> >> > >
> >> > > commit 401f56fe9d6b0271260edf9787cdcbfe4d03874d
> >> > > [FLINK-19427][FLINK-19489][tests] Fix test conditions for
> >> > > 'SplitFetcherTest.testNotifiesWhenGoingIdleConcurrent()'
> >> > >
> >> > > commit 68c5c2ff779d82a1ff81ffaf60d8a1b283797db1
> >> > > [FLINK-19448][connector base] Explicitly check for un-expected
> >> condition
> >> > > that would leave an inconsistent state
> >> > >
> >> > > commit 162c072e9265a7b6dd9d6f5459eb7974091c4c4e
> >> > > [FLINK-19492][core] Consolidate Source Events between Source API and
> >> > Split
> >> > > Reader API
> >> > >
> >> > > commit c1ca7a4c7c21ec8868c14cdddd43c559625b794c
> >> > > [refactor][tests] Move some source test utils from
> >> flink-connector-base
> >> > to
> >> > > flink-core
> >> > >
> >> > > commit ee5c4c211c35c70d28252363bbc8400453609977
> >> > > [FLINK-19251][connectors] Avoid confusing queue handling in
> >> > > "SplitReader.handleSplitsChanges()"
> >> > >
> >> > > commit 5abef56b2bf85bcac786f6b16b6899b6cced7176
> >> > > [FLINK-19250][connectors] Fix error propagation in connector base
> >> > > (SplitFetcherManager).
> >> > >
> >> > > commit 8fcca837c55a9216595ee4c03038b52747098dbb
> >> > > [hotfix][connectors] Improve JavaDocs for SingleThreadFetcherManager
> >> > >
> >> > > commit 4700bb5dde3303cbe98882f6beb7379425717b01
> >> > > [FLINK-19225][connectors] Various small improvements to
> >> SourceReaderBase
> >> > > (part 2)
> >> > >
> >> > > commit 12261c6b7ed6478a9b9f6a69cb58246b83cab9b7
> >> > > [FLINK-17393][connectors] (follow-up) Wakeup the SplitFetchers more
> >> > > elegantly.
> >> > >
> >> > > commit c60aaff0249bfd6b5871b7f82e03efc487a54d6b
> >> > > [hotfix][tests] Extend test coverage for
> FutureCompletingBlockingQueue
> >> > >
> >> > > commit cef8a587d7fd2fe64cc644da5ed095d82e46f631
> >> > > [FLINK-19245][connectors] Set default capacity for
> >> > > FutureCompletingBlockingQueue.
> >> > >
> >> > > commit 4ea95782b4c6a2538153d4d16ad3f4839c7de0fb
> >> > > [FLINK-19223][connectors] Simplify Availability Future Model in Base
> >> > > Connector
> >> > >
> >> > > commit 511857049ba30c8ff0ee56da551fa4a479dc583e
> >> > > [FLINK-18128][connectors] Ensure idle split fetchers lead to
> >> availability
> >> > > notifications.
> >> > >
> >> > > commit a8206467af0830dcb89623ea068b5ca3b3450c92
> >> > > [refactor][core] Eagerly initialize the FetchTask to support proper
> >> unit
> >> > > testing
> >> > >
> >> > > commit 3b2f54bcb437f98e6137c904045cc51072b5c06b
> >> > > [hotfix][tests] Move constants in SplitFetcherTest relevant to only
> >> one
> >> > > test into test method
> >> > >
> >> > > commit d7625760a75a508bf05bcddc380bb4d62ee1743e
> >> > > [FLINK-19225][connectors] Various small improvements to
> >> SourceReaderBase
> >> > >
> >> > > commit a5b0d3297748c1be47ad579a88f24df2255a8df1
> >> > > [FLINK-17393][connectors] Wakeup the SplitFetchers more elegantly.
> >> > >
> >> > > commit f42a3ebc3e81a034b7221a803c153636fef34903
> >> > > [FLINK-18680][connectors] Make connector base RecordsWithSplitIds
> more
> >> > > lightweight.
> >> > >
> >> > > commit e3d273de822b085183d09b275a445879ff94b350
> >> > > [FLINK-19162][connectors] Add 'recycle()' to the RecordsWithSplitIds
> >> to
> >> > > support reuse of heavy objects.
> >> > >
> >> > > commit 8ebc464c2520453a70001cd712abc8dee6ee89e0
> >> > > [hotfix][testing] Add a set of parameterizable testing mocks for the
> >> > Split
> >> > > Reader API
> >> > >
> >> > > commit 930a07438be1185388d7150640f294dfe2a5d378
> >> > > [hotfix][connectors] Add RequestSplitEvent to 'flink-connector-base'
> >> > >
> >> > > commit e945ce8a933bc378844782f784ca473c767ca159
> >> > > [refactor][connectors] Backport of the connector-base exception
> >> handling
> >> > > from the Kafka Connector Pull Request
> >> > >
> >> > >
> >> > > On Wed, Oct 28, 2020 at 5:49 PM Steven Wu <st...@gmail.com>
> >> wrote:
> >> > >
> >> > > > I would love to see this FLIP-27 source interface improvement [1]
> >> made
> >> > to
> >> > > > 1.11.3.
> >> > > >
> >> > > > [1] https://issues.apache.org/jira/browse/FLINK-19698
> >> > > >
> >> > > > On Wed, Oct 28, 2020 at 12:32 AM Tzu-Li (Gordon) Tai <
> >> > > tzulitai@apache.org>
> >> > > > wrote:
> >> > > >
> >> > > > > Thanks for the replies so far!
> >> > > > >
> >> > > > > Just to provide a brief update on the status of blockers for
> >> 1.11.3
> >> > so
> >> > > > far:
> >> > > > >
> >> > > > >
> >> > > > > *PR opened, pending reviewer*- [FLINK-19717]
> >> > SourceReaderBase.pollNext
> >> > > > may
> >> > > > > return END_OF_INPUT if SplitReader.fetch throws (
> >> > > > > https://github.com/apache/flink/pull/13776)
> >> > > > >
> >> > > > > *PR opened, reviewed + close to being merged*
> >> > > > > - [FLINK-19741] Timer service should skip restoring from raw
> keyed
> >> > > stream
> >> > > > > if it isn't the writer (
> >> https://github.com/apache/flink/pull/13761)
> >> > > > > - [FLINK-19748] Raw keyed stream key group iterator should be
> >> > skipping
> >> > > > > unwritten key groups (
> https://github.com/apache/flink/pull/13772)
> >> > > > >
> >> > > > > *Merged*
> >> > > > > - [FLINK-19154] Application mode deletes HA data in case of
> >> suspended
> >> > > > > ZooKeeper connection
> >> > > > > - [FLINK-19569] Upgrade ICU4J to 67.1+
> >> > > > >
> >> > > > > Right now as it seems, progress is mainly blocked on a reviewer
> >> for
> >> > > > > FLINK-19717.
> >> > > > > Meanwhile, Xintong is keeping an eye on test instabilities [1]
> to
> >> see
> >> > > if
> >> > > > > there are any fixes that should be applied to `release-1.11`.
> >> > > > >
> >> > > > > This is also a reminder, that if there are other blockers that
> we
> >> > need
> >> > > to
> >> > > > > be aware of, or a need to re-establish estimated time for
> getting
> >> > fixes
> >> > > > in
> >> > > > > and delay the RC for 1.11.3, please do let us know!
> >> > > > >
> >> > > > > Cheers,
> >> > > > > Gordon
> >> > > > >
> >> > > > > [1]
> >> > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
> https://issues.apache.org/jira/issues/?jql=project%20%3D%20FLINK%20AND%20status%20in%20(Open%2C%20%22In%20Progress%22%2C%20Reopened)%20AND%20affectedVersion%20in%20(1.11.0%2C%201.11.1%2C%201.11.2)%20AND%20labels%20%3D%20test-stability
> >> > > > >
> >> > > > > On Mon, Oct 26, 2020 at 9:43 PM Kostas Kloudas <
> >> kkloudas@apache.org>
> >> > > > > wrote:
> >> > > > >
> >> > > > > > +1 for releasing Flink 1.11.3 as it contains a number of
> >> important
> >> > > > > > fixes and thanks Gordon and Xintong for volunteering.
> >> > > > > >
> >> > > > > > Cheers,
> >> > > > > > Kostas
> >> > > > > >
> >> > > > > > On Mon, Oct 26, 2020 at 4:37 AM Yu Li <ca...@gmail.com>
> wrote:
> >> > > > > > >
> >> > > > > > > +1 for releasing Flink 1.11.3, and thanks Gordon and Xintong
> >> for
> >> > > > > > > volunteering as our release managers.
> >> > > > > > >
> >> > > > > > > Best Regards,
> >> > > > > > > Yu
> >> > > > > > >
> >> > > > > > >
> >> > > > > > > On Mon, 26 Oct 2020 at 09:45, Xintong Song <
> >> > tonysong820@gmail.com>
> >> > > > > > wrote:
> >> > > > > > >
> >> > > > > > > > Thanks Gordan for starting this discussion.
> >> > > > > > > > My pleasure to help with the release process.
> >> > > > > > > >
> >> > > > > > > > Thank you~
> >> > > > > > > >
> >> > > > > > > > Xintong Song
> >> > > > > > > >
> >> > > > > > > >
> >> > > > > > > >
> >> > > > > > > > On Fri, Oct 23, 2020 at 11:29 PM Till Rohrmann <
> >> > > > trohrmann@apache.org
> >> > > > > >
> >> > > > > > > > wrote:
> >> > > > > > > >
> >> > > > > > > > > Thanks for starting this discussion Gordon. There are
> over
> >> > 100
> >> > > > > issues
> >> > > > > > > > > which are fixed for 1.11.3. Hence +1 for a soonish
> 1.11.3
> >> > > > release.
> >> > > > > > Thanks
> >> > > > > > > > > for volunteering as our release managers Gordon and
> >> Xintong!
> >> > > > > > > > >
> >> > > > > > > > > Cheers,
> >> > > > > > > > > Till
> >> > > > > > > > >
> >> > > > > > > > > On Fri, Oct 23, 2020 at 5:02 PM Tzu-Li (Gordon) Tai <
> >> > > > > > tzulitai@apache.org
> >> > > > > > > > >
> >> > > > > > > > > wrote:
> >> > > > > > > > >
> >> > > > > > > > >> Hi,
> >> > > > > > > > >>
> >> > > > > > > > >> Xintong and I would like to start a discussion for
> >> releasing
> >> > > > Flink
> >> > > > > > > > 1.11.3
> >> > > > > > > > >> soon.
> >> > > > > > > > >>
> >> > > > > > > > >> It seems like we already have a few pressing issues
> that
> >> > needs
> >> > > > to
> >> > > > > be
> >> > > > > > > > >> included in a new hotfix release:
> >> > > > > > > > >>
> >> > > > > > > > >>    - Heap-based timers’ restore behaviour is causing a
> >> > > critical
> >> > > > > > recovery
> >> > > > > > > > >>    issue for StateFun [1] [2] [3].
> >> > > > > > > > >>    - There are several robustness issues for the
> FLIP-27
> >> new
> >> > > > > source
> >> > > > > > API,
> >> > > > > > > > >>    such as [4]. We already have some users using the
> >> FLIP-27
> >> > > API
> >> > > > > > with
> >> > > > > > > > >> 1.11.x,
> >> > > > > > > > >>    so it would be important to get those fixes in for
> >> 1.11.x
> >> > > as
> >> > > > > > well.
> >> > > > > > > > >>
> >> > > > > > > > >> Apart from the issues that are already marked as
> blocker
> >> for
> >> > > > > 1.11.3
> >> > > > > > in
> >> > > > > > > > our
> >> > > > > > > > >> JIRA [5], please let us know in this thread if there is
> >> > > already
> >> > > > > > ongoing
> >> > > > > > > > >> work for other important fixes that we should try to
> >> > include.
> >> > > > > > > > >>
> >> > > > > > > > >> Xintong and I would like to volunteer for managing this
> >> > > release,
> >> > > > > and
> >> > > > > > > > will
> >> > > > > > > > >> try to communicate the priority of pending blockers
> over
> >> the
> >> > > > next
> >> > > > > > few
> >> > > > > > > > >> days.
> >> > > > > > > > >> Since the aforementioned issues are quite critical,
> we’d
> >> > like
> >> > > to
> >> > > > > aim
> >> > > > > > > > >> for a *feature
> >> > > > > > > > >> freeze by the end of next week (Oct. 30th)* and start
> the
> >> > > > release
> >> > > > > > voting
> >> > > > > > > > >> process the week after.
> >> > > > > > > > >> If that is too short of a notice and you might need
> more
> >> > time,
> >> > > > > > please
> >> > > > > > > > let
> >> > > > > > > > >> us know!
> >> > > > > > > > >>
> >> > > > > > > > >> Cheers,
> >> > > > > > > > >> Gordon
> >> > > > > > > > >>
> >> > > > > > > > >> [1] https://issues.apache.org/jira/browse/FLINK-19692
> >> > > > > > > > >> [2] https://issues.apache.org/jira/browse/FLINK-19741
> >> > > > > > > > >> [3] https://issues.apache.org/jira/browse/FLINK-19748
> >> > > > > > > > >> [4] https://issues.apache.org/jira/browse/FLINK-19717
> >> > > > > > > > >> [5]
> >> > > > > > > > >>
> >> > > > > > > > >>
> >> > > > > > > >
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
> https://issues.apache.org/jira/issues/?jql=project%20%3D%20FLINK%20AND%20priority%20%3D%20Blocker%20AND%20fixVersion%20%3D%201.11.3
> >> > > > > > > > >>
> >> > > > > > > > >
> >> > > > > > > >
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >> >
> >> > --
> >> > Best, Jingsong Lee
> >> >
> >>
> >
>

Re: [DISCUSS] Releasing Apache Flink 1.11.3

Posted by Stephan Ewen <se...@apache.org>.
I have started with backporting the source API changes. Some minor
conflicts to solve, will need a bit more to finish this.

On Fri, Oct 30, 2020 at 7:25 AM Tzu-Li (Gordon) Tai <tz...@apache.org>
wrote:

> @Stephan Ewen <se...@apache.org>
> Are there already plans or ongoing efforts for backporting the list of
> FLIP-27 changes that you posted?
>
> On Thu, Oct 29, 2020 at 7:08 PM Xintong Song <to...@gmail.com>
> wrote:
>
>> Hi folks,
>>
>> Just to provide some updates concerning the status on the
>> test instabilities.
>>
>> Currently, we have 30 unresolved tickets labeled with `Affects Version`
>> 1.11.x.
>>
>> https://issues.apache.org/jira/browse/FLINK-19775?filter=12348580&jql=project%20%3D%20FLINK%20AND%20resolution%20%3D%20Unresolved%20AND%20affectedVersion%20in%20(1.11.0%2C%201.11.1%2C%201.11.2%2C%201.11.3)%20AND%20labels%20%3D%20test-stability%20ORDER%20BY%20created%20DESC
>>
>> Among the 30 tickets, 11 of them are:
>> - Have occured in the recent 3 months
>> - Not confirmed to be pure testability issues
>> - Not confirmed to be rare condition cases
>>
>> It would be nice if someone familiar with these components can take a look
>> into these issues.
>>
>> - https://issues.apache.org/jira/browse/FLINK-17159 (ES6)
>> - https://issues.apache.org/jira/browse/FLINK-17912 (Kafka)
>> - https://issues.apache.org/jira/browse/FLINK-17949 (Kafka)
>> ⁃ https://issues.apache.org/jira/browse/FLINK-18444 (Kafka)
>> - https://issues.apache.org/jira/browse/FLINK-18634 (Kafka)
>> - https://issues.apache.org/jira/browse/FLINK-18648 (Kafka)
>> - https://issues.apache.org/jira/browse/FLINK-18807 (Kafka)
>> - https://issues.apache.org/jira/browse/FLINK-19369 (BlobClientTest)
>> - https://issues.apache.org/jira/browse/FLINK-19436 (TPCDS)
>> - https://issues.apache.org/jira/browse/FLINK-19690 (Format/Parquet)
>> - https://issues.apache.org/jira/browse/FLINK-19775
>> (SystemProcessingTimeServiceTest)
>>
>> Thank you~
>>
>> Xintong Song
>>
>>
>>
>> On Thu, Oct 29, 2020 at 10:21 AM Jingsong Li <ji...@gmail.com>
>> wrote:
>>
>> > +1 to backport the FLIP-27 adjustments to 1.11.x.
>> >
>> > If possible, that would be great. Many people are looking forward to the
>> > FLIP-27 interface, but they don't want to take the risk to upgrade to
>> 1.12
>> > (And wait 1.12). After all, 1.11 is a relatively stable version.
>> >
>> > Best,
>> > Jingsong
>> >
>> > On Thu, Oct 29, 2020 at 1:24 AM Stephan Ewen <se...@apache.org> wrote:
>> >
>> > > Thanks for starting this.
>> > >
>> > > +1 form my side to backport the FLIP-27 adjustments to 1.11.x.
>> > >
>> > > There were quite a few changes, and I think we need to cherry-pick
>> them
>> > all
>> > > to not get some inconsistent mix of changes and many merge conflicts.
>> > > I made a list below of what we need to add to "release-1.11".
>> > >
>> > > *#### Core Source API Changes to backport (in REVERSE order)*
>> > >
>> > >   (Use: "git log
>> > > flink-core/src/main/java/org/apache/flink/api/connector/source")
>> > >
>> > > commit 162c072e9265a7b6dd9d6f5459eb7974091c4c4e
>> > > [FLINK-19492][core] Consolidate Source Events between Source API and
>> > Split
>> > > Reader API
>> > >
>> > > commit 428fbbad85379f71a9fbf1a1ab3e3dcfa9006436
>> > > [FLINK-19205][core] Add access to configuration and hostname in the
>> > > SourceReaderContext
>> > >
>> > > commit 723e1790744ddc78d19c7c978442af1383f38d33
>> > > [hotfix][core] Add to Source Enumerator convenience methods to assign
>> > > single split
>> > >
>> > > *#### Connector Base Changes to Backport (in REVERSE order)*
>> > >
>> > >   (Use: "git log flink-connectors/flink-connector-base")
>> > >
>> > > commit 401f56fe9d6b0271260edf9787cdcbfe4d03874d
>> > > [FLINK-19427][FLINK-19489][tests] Fix test conditions for
>> > > 'SplitFetcherTest.testNotifiesWhenGoingIdleConcurrent()'
>> > >
>> > > commit 68c5c2ff779d82a1ff81ffaf60d8a1b283797db1
>> > > [FLINK-19448][connector base] Explicitly check for un-expected
>> condition
>> > > that would leave an inconsistent state
>> > >
>> > > commit 162c072e9265a7b6dd9d6f5459eb7974091c4c4e
>> > > [FLINK-19492][core] Consolidate Source Events between Source API and
>> > Split
>> > > Reader API
>> > >
>> > > commit c1ca7a4c7c21ec8868c14cdddd43c559625b794c
>> > > [refactor][tests] Move some source test utils from
>> flink-connector-base
>> > to
>> > > flink-core
>> > >
>> > > commit ee5c4c211c35c70d28252363bbc8400453609977
>> > > [FLINK-19251][connectors] Avoid confusing queue handling in
>> > > "SplitReader.handleSplitsChanges()"
>> > >
>> > > commit 5abef56b2bf85bcac786f6b16b6899b6cced7176
>> > > [FLINK-19250][connectors] Fix error propagation in connector base
>> > > (SplitFetcherManager).
>> > >
>> > > commit 8fcca837c55a9216595ee4c03038b52747098dbb
>> > > [hotfix][connectors] Improve JavaDocs for SingleThreadFetcherManager
>> > >
>> > > commit 4700bb5dde3303cbe98882f6beb7379425717b01
>> > > [FLINK-19225][connectors] Various small improvements to
>> SourceReaderBase
>> > > (part 2)
>> > >
>> > > commit 12261c6b7ed6478a9b9f6a69cb58246b83cab9b7
>> > > [FLINK-17393][connectors] (follow-up) Wakeup the SplitFetchers more
>> > > elegantly.
>> > >
>> > > commit c60aaff0249bfd6b5871b7f82e03efc487a54d6b
>> > > [hotfix][tests] Extend test coverage for FutureCompletingBlockingQueue
>> > >
>> > > commit cef8a587d7fd2fe64cc644da5ed095d82e46f631
>> > > [FLINK-19245][connectors] Set default capacity for
>> > > FutureCompletingBlockingQueue.
>> > >
>> > > commit 4ea95782b4c6a2538153d4d16ad3f4839c7de0fb
>> > > [FLINK-19223][connectors] Simplify Availability Future Model in Base
>> > > Connector
>> > >
>> > > commit 511857049ba30c8ff0ee56da551fa4a479dc583e
>> > > [FLINK-18128][connectors] Ensure idle split fetchers lead to
>> availability
>> > > notifications.
>> > >
>> > > commit a8206467af0830dcb89623ea068b5ca3b3450c92
>> > > [refactor][core] Eagerly initialize the FetchTask to support proper
>> unit
>> > > testing
>> > >
>> > > commit 3b2f54bcb437f98e6137c904045cc51072b5c06b
>> > > [hotfix][tests] Move constants in SplitFetcherTest relevant to only
>> one
>> > > test into test method
>> > >
>> > > commit d7625760a75a508bf05bcddc380bb4d62ee1743e
>> > > [FLINK-19225][connectors] Various small improvements to
>> SourceReaderBase
>> > >
>> > > commit a5b0d3297748c1be47ad579a88f24df2255a8df1
>> > > [FLINK-17393][connectors] Wakeup the SplitFetchers more elegantly.
>> > >
>> > > commit f42a3ebc3e81a034b7221a803c153636fef34903
>> > > [FLINK-18680][connectors] Make connector base RecordsWithSplitIds more
>> > > lightweight.
>> > >
>> > > commit e3d273de822b085183d09b275a445879ff94b350
>> > > [FLINK-19162][connectors] Add 'recycle()' to the RecordsWithSplitIds
>> to
>> > > support reuse of heavy objects.
>> > >
>> > > commit 8ebc464c2520453a70001cd712abc8dee6ee89e0
>> > > [hotfix][testing] Add a set of parameterizable testing mocks for the
>> > Split
>> > > Reader API
>> > >
>> > > commit 930a07438be1185388d7150640f294dfe2a5d378
>> > > [hotfix][connectors] Add RequestSplitEvent to 'flink-connector-base'
>> > >
>> > > commit e945ce8a933bc378844782f784ca473c767ca159
>> > > [refactor][connectors] Backport of the connector-base exception
>> handling
>> > > from the Kafka Connector Pull Request
>> > >
>> > >
>> > > On Wed, Oct 28, 2020 at 5:49 PM Steven Wu <st...@gmail.com>
>> wrote:
>> > >
>> > > > I would love to see this FLIP-27 source interface improvement [1]
>> made
>> > to
>> > > > 1.11.3.
>> > > >
>> > > > [1] https://issues.apache.org/jira/browse/FLINK-19698
>> > > >
>> > > > On Wed, Oct 28, 2020 at 12:32 AM Tzu-Li (Gordon) Tai <
>> > > tzulitai@apache.org>
>> > > > wrote:
>> > > >
>> > > > > Thanks for the replies so far!
>> > > > >
>> > > > > Just to provide a brief update on the status of blockers for
>> 1.11.3
>> > so
>> > > > far:
>> > > > >
>> > > > >
>> > > > > *PR opened, pending reviewer*- [FLINK-19717]
>> > SourceReaderBase.pollNext
>> > > > may
>> > > > > return END_OF_INPUT if SplitReader.fetch throws (
>> > > > > https://github.com/apache/flink/pull/13776)
>> > > > >
>> > > > > *PR opened, reviewed + close to being merged*
>> > > > > - [FLINK-19741] Timer service should skip restoring from raw keyed
>> > > stream
>> > > > > if it isn't the writer (
>> https://github.com/apache/flink/pull/13761)
>> > > > > - [FLINK-19748] Raw keyed stream key group iterator should be
>> > skipping
>> > > > > unwritten key groups (https://github.com/apache/flink/pull/13772)
>> > > > >
>> > > > > *Merged*
>> > > > > - [FLINK-19154] Application mode deletes HA data in case of
>> suspended
>> > > > > ZooKeeper connection
>> > > > > - [FLINK-19569] Upgrade ICU4J to 67.1+
>> > > > >
>> > > > > Right now as it seems, progress is mainly blocked on a reviewer
>> for
>> > > > > FLINK-19717.
>> > > > > Meanwhile, Xintong is keeping an eye on test instabilities [1] to
>> see
>> > > if
>> > > > > there are any fixes that should be applied to `release-1.11`.
>> > > > >
>> > > > > This is also a reminder, that if there are other blockers that we
>> > need
>> > > to
>> > > > > be aware of, or a need to re-establish estimated time for getting
>> > fixes
>> > > > in
>> > > > > and delay the RC for 1.11.3, please do let us know!
>> > > > >
>> > > > > Cheers,
>> > > > > Gordon
>> > > > >
>> > > > > [1]
>> > > > >
>> > > > >
>> > > >
>> > >
>> >
>> https://issues.apache.org/jira/issues/?jql=project%20%3D%20FLINK%20AND%20status%20in%20(Open%2C%20%22In%20Progress%22%2C%20Reopened)%20AND%20affectedVersion%20in%20(1.11.0%2C%201.11.1%2C%201.11.2)%20AND%20labels%20%3D%20test-stability
>> > > > >
>> > > > > On Mon, Oct 26, 2020 at 9:43 PM Kostas Kloudas <
>> kkloudas@apache.org>
>> > > > > wrote:
>> > > > >
>> > > > > > +1 for releasing Flink 1.11.3 as it contains a number of
>> important
>> > > > > > fixes and thanks Gordon and Xintong for volunteering.
>> > > > > >
>> > > > > > Cheers,
>> > > > > > Kostas
>> > > > > >
>> > > > > > On Mon, Oct 26, 2020 at 4:37 AM Yu Li <ca...@gmail.com> wrote:
>> > > > > > >
>> > > > > > > +1 for releasing Flink 1.11.3, and thanks Gordon and Xintong
>> for
>> > > > > > > volunteering as our release managers.
>> > > > > > >
>> > > > > > > Best Regards,
>> > > > > > > Yu
>> > > > > > >
>> > > > > > >
>> > > > > > > On Mon, 26 Oct 2020 at 09:45, Xintong Song <
>> > tonysong820@gmail.com>
>> > > > > > wrote:
>> > > > > > >
>> > > > > > > > Thanks Gordan for starting this discussion.
>> > > > > > > > My pleasure to help with the release process.
>> > > > > > > >
>> > > > > > > > Thank you~
>> > > > > > > >
>> > > > > > > > Xintong Song
>> > > > > > > >
>> > > > > > > >
>> > > > > > > >
>> > > > > > > > On Fri, Oct 23, 2020 at 11:29 PM Till Rohrmann <
>> > > > trohrmann@apache.org
>> > > > > >
>> > > > > > > > wrote:
>> > > > > > > >
>> > > > > > > > > Thanks for starting this discussion Gordon. There are over
>> > 100
>> > > > > issues
>> > > > > > > > > which are fixed for 1.11.3. Hence +1 for a soonish 1.11.3
>> > > > release.
>> > > > > > Thanks
>> > > > > > > > > for volunteering as our release managers Gordon and
>> Xintong!
>> > > > > > > > >
>> > > > > > > > > Cheers,
>> > > > > > > > > Till
>> > > > > > > > >
>> > > > > > > > > On Fri, Oct 23, 2020 at 5:02 PM Tzu-Li (Gordon) Tai <
>> > > > > > tzulitai@apache.org
>> > > > > > > > >
>> > > > > > > > > wrote:
>> > > > > > > > >
>> > > > > > > > >> Hi,
>> > > > > > > > >>
>> > > > > > > > >> Xintong and I would like to start a discussion for
>> releasing
>> > > > Flink
>> > > > > > > > 1.11.3
>> > > > > > > > >> soon.
>> > > > > > > > >>
>> > > > > > > > >> It seems like we already have a few pressing issues that
>> > needs
>> > > > to
>> > > > > be
>> > > > > > > > >> included in a new hotfix release:
>> > > > > > > > >>
>> > > > > > > > >>    - Heap-based timers’ restore behaviour is causing a
>> > > critical
>> > > > > > recovery
>> > > > > > > > >>    issue for StateFun [1] [2] [3].
>> > > > > > > > >>    - There are several robustness issues for the FLIP-27
>> new
>> > > > > source
>> > > > > > API,
>> > > > > > > > >>    such as [4]. We already have some users using the
>> FLIP-27
>> > > API
>> > > > > > with
>> > > > > > > > >> 1.11.x,
>> > > > > > > > >>    so it would be important to get those fixes in for
>> 1.11.x
>> > > as
>> > > > > > well.
>> > > > > > > > >>
>> > > > > > > > >> Apart from the issues that are already marked as blocker
>> for
>> > > > > 1.11.3
>> > > > > > in
>> > > > > > > > our
>> > > > > > > > >> JIRA [5], please let us know in this thread if there is
>> > > already
>> > > > > > ongoing
>> > > > > > > > >> work for other important fixes that we should try to
>> > include.
>> > > > > > > > >>
>> > > > > > > > >> Xintong and I would like to volunteer for managing this
>> > > release,
>> > > > > and
>> > > > > > > > will
>> > > > > > > > >> try to communicate the priority of pending blockers over
>> the
>> > > > next
>> > > > > > few
>> > > > > > > > >> days.
>> > > > > > > > >> Since the aforementioned issues are quite critical, we’d
>> > like
>> > > to
>> > > > > aim
>> > > > > > > > >> for a *feature
>> > > > > > > > >> freeze by the end of next week (Oct. 30th)* and start the
>> > > > release
>> > > > > > voting
>> > > > > > > > >> process the week after.
>> > > > > > > > >> If that is too short of a notice and you might need more
>> > time,
>> > > > > > please
>> > > > > > > > let
>> > > > > > > > >> us know!
>> > > > > > > > >>
>> > > > > > > > >> Cheers,
>> > > > > > > > >> Gordon
>> > > > > > > > >>
>> > > > > > > > >> [1] https://issues.apache.org/jira/browse/FLINK-19692
>> > > > > > > > >> [2] https://issues.apache.org/jira/browse/FLINK-19741
>> > > > > > > > >> [3] https://issues.apache.org/jira/browse/FLINK-19748
>> > > > > > > > >> [4] https://issues.apache.org/jira/browse/FLINK-19717
>> > > > > > > > >> [5]
>> > > > > > > > >>
>> > > > > > > > >>
>> > > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> https://issues.apache.org/jira/issues/?jql=project%20%3D%20FLINK%20AND%20priority%20%3D%20Blocker%20AND%20fixVersion%20%3D%201.11.3
>> > > > > > > > >>
>> > > > > > > > >
>> > > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> >
>> > --
>> > Best, Jingsong Lee
>> >
>>
>

Re: [DISCUSS] Releasing Apache Flink 1.11.3

Posted by "Tzu-Li (Gordon) Tai" <tz...@apache.org>.
@Stephan Ewen <se...@apache.org>
Are there already plans or ongoing efforts for backporting the list of
FLIP-27 changes that you posted?

On Thu, Oct 29, 2020 at 7:08 PM Xintong Song <to...@gmail.com> wrote:

> Hi folks,
>
> Just to provide some updates concerning the status on the
> test instabilities.
>
> Currently, we have 30 unresolved tickets labeled with `Affects Version`
> 1.11.x.
>
> https://issues.apache.org/jira/browse/FLINK-19775?filter=12348580&jql=project%20%3D%20FLINK%20AND%20resolution%20%3D%20Unresolved%20AND%20affectedVersion%20in%20(1.11.0%2C%201.11.1%2C%201.11.2%2C%201.11.3)%20AND%20labels%20%3D%20test-stability%20ORDER%20BY%20created%20DESC
>
> Among the 30 tickets, 11 of them are:
> - Have occured in the recent 3 months
> - Not confirmed to be pure testability issues
> - Not confirmed to be rare condition cases
>
> It would be nice if someone familiar with these components can take a look
> into these issues.
>
> - https://issues.apache.org/jira/browse/FLINK-17159 (ES6)
> - https://issues.apache.org/jira/browse/FLINK-17912 (Kafka)
> - https://issues.apache.org/jira/browse/FLINK-17949 (Kafka)
> ⁃ https://issues.apache.org/jira/browse/FLINK-18444 (Kafka)
> - https://issues.apache.org/jira/browse/FLINK-18634 (Kafka)
> - https://issues.apache.org/jira/browse/FLINK-18648 (Kafka)
> - https://issues.apache.org/jira/browse/FLINK-18807 (Kafka)
> - https://issues.apache.org/jira/browse/FLINK-19369 (BlobClientTest)
> - https://issues.apache.org/jira/browse/FLINK-19436 (TPCDS)
> - https://issues.apache.org/jira/browse/FLINK-19690 (Format/Parquet)
> - https://issues.apache.org/jira/browse/FLINK-19775
> (SystemProcessingTimeServiceTest)
>
> Thank you~
>
> Xintong Song
>
>
>
> On Thu, Oct 29, 2020 at 10:21 AM Jingsong Li <ji...@gmail.com>
> wrote:
>
> > +1 to backport the FLIP-27 adjustments to 1.11.x.
> >
> > If possible, that would be great. Many people are looking forward to the
> > FLIP-27 interface, but they don't want to take the risk to upgrade to
> 1.12
> > (And wait 1.12). After all, 1.11 is a relatively stable version.
> >
> > Best,
> > Jingsong
> >
> > On Thu, Oct 29, 2020 at 1:24 AM Stephan Ewen <se...@apache.org> wrote:
> >
> > > Thanks for starting this.
> > >
> > > +1 form my side to backport the FLIP-27 adjustments to 1.11.x.
> > >
> > > There were quite a few changes, and I think we need to cherry-pick them
> > all
> > > to not get some inconsistent mix of changes and many merge conflicts.
> > > I made a list below of what we need to add to "release-1.11".
> > >
> > > *#### Core Source API Changes to backport (in REVERSE order)*
> > >
> > >   (Use: "git log
> > > flink-core/src/main/java/org/apache/flink/api/connector/source")
> > >
> > > commit 162c072e9265a7b6dd9d6f5459eb7974091c4c4e
> > > [FLINK-19492][core] Consolidate Source Events between Source API and
> > Split
> > > Reader API
> > >
> > > commit 428fbbad85379f71a9fbf1a1ab3e3dcfa9006436
> > > [FLINK-19205][core] Add access to configuration and hostname in the
> > > SourceReaderContext
> > >
> > > commit 723e1790744ddc78d19c7c978442af1383f38d33
> > > [hotfix][core] Add to Source Enumerator convenience methods to assign
> > > single split
> > >
> > > *#### Connector Base Changes to Backport (in REVERSE order)*
> > >
> > >   (Use: "git log flink-connectors/flink-connector-base")
> > >
> > > commit 401f56fe9d6b0271260edf9787cdcbfe4d03874d
> > > [FLINK-19427][FLINK-19489][tests] Fix test conditions for
> > > 'SplitFetcherTest.testNotifiesWhenGoingIdleConcurrent()'
> > >
> > > commit 68c5c2ff779d82a1ff81ffaf60d8a1b283797db1
> > > [FLINK-19448][connector base] Explicitly check for un-expected
> condition
> > > that would leave an inconsistent state
> > >
> > > commit 162c072e9265a7b6dd9d6f5459eb7974091c4c4e
> > > [FLINK-19492][core] Consolidate Source Events between Source API and
> > Split
> > > Reader API
> > >
> > > commit c1ca7a4c7c21ec8868c14cdddd43c559625b794c
> > > [refactor][tests] Move some source test utils from flink-connector-base
> > to
> > > flink-core
> > >
> > > commit ee5c4c211c35c70d28252363bbc8400453609977
> > > [FLINK-19251][connectors] Avoid confusing queue handling in
> > > "SplitReader.handleSplitsChanges()"
> > >
> > > commit 5abef56b2bf85bcac786f6b16b6899b6cced7176
> > > [FLINK-19250][connectors] Fix error propagation in connector base
> > > (SplitFetcherManager).
> > >
> > > commit 8fcca837c55a9216595ee4c03038b52747098dbb
> > > [hotfix][connectors] Improve JavaDocs for SingleThreadFetcherManager
> > >
> > > commit 4700bb5dde3303cbe98882f6beb7379425717b01
> > > [FLINK-19225][connectors] Various small improvements to
> SourceReaderBase
> > > (part 2)
> > >
> > > commit 12261c6b7ed6478a9b9f6a69cb58246b83cab9b7
> > > [FLINK-17393][connectors] (follow-up) Wakeup the SplitFetchers more
> > > elegantly.
> > >
> > > commit c60aaff0249bfd6b5871b7f82e03efc487a54d6b
> > > [hotfix][tests] Extend test coverage for FutureCompletingBlockingQueue
> > >
> > > commit cef8a587d7fd2fe64cc644da5ed095d82e46f631
> > > [FLINK-19245][connectors] Set default capacity for
> > > FutureCompletingBlockingQueue.
> > >
> > > commit 4ea95782b4c6a2538153d4d16ad3f4839c7de0fb
> > > [FLINK-19223][connectors] Simplify Availability Future Model in Base
> > > Connector
> > >
> > > commit 511857049ba30c8ff0ee56da551fa4a479dc583e
> > > [FLINK-18128][connectors] Ensure idle split fetchers lead to
> availability
> > > notifications.
> > >
> > > commit a8206467af0830dcb89623ea068b5ca3b3450c92
> > > [refactor][core] Eagerly initialize the FetchTask to support proper
> unit
> > > testing
> > >
> > > commit 3b2f54bcb437f98e6137c904045cc51072b5c06b
> > > [hotfix][tests] Move constants in SplitFetcherTest relevant to only one
> > > test into test method
> > >
> > > commit d7625760a75a508bf05bcddc380bb4d62ee1743e
> > > [FLINK-19225][connectors] Various small improvements to
> SourceReaderBase
> > >
> > > commit a5b0d3297748c1be47ad579a88f24df2255a8df1
> > > [FLINK-17393][connectors] Wakeup the SplitFetchers more elegantly.
> > >
> > > commit f42a3ebc3e81a034b7221a803c153636fef34903
> > > [FLINK-18680][connectors] Make connector base RecordsWithSplitIds more
> > > lightweight.
> > >
> > > commit e3d273de822b085183d09b275a445879ff94b350
> > > [FLINK-19162][connectors] Add 'recycle()' to the RecordsWithSplitIds to
> > > support reuse of heavy objects.
> > >
> > > commit 8ebc464c2520453a70001cd712abc8dee6ee89e0
> > > [hotfix][testing] Add a set of parameterizable testing mocks for the
> > Split
> > > Reader API
> > >
> > > commit 930a07438be1185388d7150640f294dfe2a5d378
> > > [hotfix][connectors] Add RequestSplitEvent to 'flink-connector-base'
> > >
> > > commit e945ce8a933bc378844782f784ca473c767ca159
> > > [refactor][connectors] Backport of the connector-base exception
> handling
> > > from the Kafka Connector Pull Request
> > >
> > >
> > > On Wed, Oct 28, 2020 at 5:49 PM Steven Wu <st...@gmail.com>
> wrote:
> > >
> > > > I would love to see this FLIP-27 source interface improvement [1]
> made
> > to
> > > > 1.11.3.
> > > >
> > > > [1] https://issues.apache.org/jira/browse/FLINK-19698
> > > >
> > > > On Wed, Oct 28, 2020 at 12:32 AM Tzu-Li (Gordon) Tai <
> > > tzulitai@apache.org>
> > > > wrote:
> > > >
> > > > > Thanks for the replies so far!
> > > > >
> > > > > Just to provide a brief update on the status of blockers for 1.11.3
> > so
> > > > far:
> > > > >
> > > > >
> > > > > *PR opened, pending reviewer*- [FLINK-19717]
> > SourceReaderBase.pollNext
> > > > may
> > > > > return END_OF_INPUT if SplitReader.fetch throws (
> > > > > https://github.com/apache/flink/pull/13776)
> > > > >
> > > > > *PR opened, reviewed + close to being merged*
> > > > > - [FLINK-19741] Timer service should skip restoring from raw keyed
> > > stream
> > > > > if it isn't the writer (https://github.com/apache/flink/pull/13761
> )
> > > > > - [FLINK-19748] Raw keyed stream key group iterator should be
> > skipping
> > > > > unwritten key groups (https://github.com/apache/flink/pull/13772)
> > > > >
> > > > > *Merged*
> > > > > - [FLINK-19154] Application mode deletes HA data in case of
> suspended
> > > > > ZooKeeper connection
> > > > > - [FLINK-19569] Upgrade ICU4J to 67.1+
> > > > >
> > > > > Right now as it seems, progress is mainly blocked on a reviewer for
> > > > > FLINK-19717.
> > > > > Meanwhile, Xintong is keeping an eye on test instabilities [1] to
> see
> > > if
> > > > > there are any fixes that should be applied to `release-1.11`.
> > > > >
> > > > > This is also a reminder, that if there are other blockers that we
> > need
> > > to
> > > > > be aware of, or a need to re-establish estimated time for getting
> > fixes
> > > > in
> > > > > and delay the RC for 1.11.3, please do let us know!
> > > > >
> > > > > Cheers,
> > > > > Gordon
> > > > >
> > > > > [1]
> > > > >
> > > > >
> > > >
> > >
> >
> https://issues.apache.org/jira/issues/?jql=project%20%3D%20FLINK%20AND%20status%20in%20(Open%2C%20%22In%20Progress%22%2C%20Reopened)%20AND%20affectedVersion%20in%20(1.11.0%2C%201.11.1%2C%201.11.2)%20AND%20labels%20%3D%20test-stability
> > > > >
> > > > > On Mon, Oct 26, 2020 at 9:43 PM Kostas Kloudas <
> kkloudas@apache.org>
> > > > > wrote:
> > > > >
> > > > > > +1 for releasing Flink 1.11.3 as it contains a number of
> important
> > > > > > fixes and thanks Gordon and Xintong for volunteering.
> > > > > >
> > > > > > Cheers,
> > > > > > Kostas
> > > > > >
> > > > > > On Mon, Oct 26, 2020 at 4:37 AM Yu Li <ca...@gmail.com> wrote:
> > > > > > >
> > > > > > > +1 for releasing Flink 1.11.3, and thanks Gordon and Xintong
> for
> > > > > > > volunteering as our release managers.
> > > > > > >
> > > > > > > Best Regards,
> > > > > > > Yu
> > > > > > >
> > > > > > >
> > > > > > > On Mon, 26 Oct 2020 at 09:45, Xintong Song <
> > tonysong820@gmail.com>
> > > > > > wrote:
> > > > > > >
> > > > > > > > Thanks Gordan for starting this discussion.
> > > > > > > > My pleasure to help with the release process.
> > > > > > > >
> > > > > > > > Thank you~
> > > > > > > >
> > > > > > > > Xintong Song
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > On Fri, Oct 23, 2020 at 11:29 PM Till Rohrmann <
> > > > trohrmann@apache.org
> > > > > >
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Thanks for starting this discussion Gordon. There are over
> > 100
> > > > > issues
> > > > > > > > > which are fixed for 1.11.3. Hence +1 for a soonish 1.11.3
> > > > release.
> > > > > > Thanks
> > > > > > > > > for volunteering as our release managers Gordon and
> Xintong!
> > > > > > > > >
> > > > > > > > > Cheers,
> > > > > > > > > Till
> > > > > > > > >
> > > > > > > > > On Fri, Oct 23, 2020 at 5:02 PM Tzu-Li (Gordon) Tai <
> > > > > > tzulitai@apache.org
> > > > > > > > >
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > >> Hi,
> > > > > > > > >>
> > > > > > > > >> Xintong and I would like to start a discussion for
> releasing
> > > > Flink
> > > > > > > > 1.11.3
> > > > > > > > >> soon.
> > > > > > > > >>
> > > > > > > > >> It seems like we already have a few pressing issues that
> > needs
> > > > to
> > > > > be
> > > > > > > > >> included in a new hotfix release:
> > > > > > > > >>
> > > > > > > > >>    - Heap-based timers’ restore behaviour is causing a
> > > critical
> > > > > > recovery
> > > > > > > > >>    issue for StateFun [1] [2] [3].
> > > > > > > > >>    - There are several robustness issues for the FLIP-27
> new
> > > > > source
> > > > > > API,
> > > > > > > > >>    such as [4]. We already have some users using the
> FLIP-27
> > > API
> > > > > > with
> > > > > > > > >> 1.11.x,
> > > > > > > > >>    so it would be important to get those fixes in for
> 1.11.x
> > > as
> > > > > > well.
> > > > > > > > >>
> > > > > > > > >> Apart from the issues that are already marked as blocker
> for
> > > > > 1.11.3
> > > > > > in
> > > > > > > > our
> > > > > > > > >> JIRA [5], please let us know in this thread if there is
> > > already
> > > > > > ongoing
> > > > > > > > >> work for other important fixes that we should try to
> > include.
> > > > > > > > >>
> > > > > > > > >> Xintong and I would like to volunteer for managing this
> > > release,
> > > > > and
> > > > > > > > will
> > > > > > > > >> try to communicate the priority of pending blockers over
> the
> > > > next
> > > > > > few
> > > > > > > > >> days.
> > > > > > > > >> Since the aforementioned issues are quite critical, we’d
> > like
> > > to
> > > > > aim
> > > > > > > > >> for a *feature
> > > > > > > > >> freeze by the end of next week (Oct. 30th)* and start the
> > > > release
> > > > > > voting
> > > > > > > > >> process the week after.
> > > > > > > > >> If that is too short of a notice and you might need more
> > time,
> > > > > > please
> > > > > > > > let
> > > > > > > > >> us know!
> > > > > > > > >>
> > > > > > > > >> Cheers,
> > > > > > > > >> Gordon
> > > > > > > > >>
> > > > > > > > >> [1] https://issues.apache.org/jira/browse/FLINK-19692
> > > > > > > > >> [2] https://issues.apache.org/jira/browse/FLINK-19741
> > > > > > > > >> [3] https://issues.apache.org/jira/browse/FLINK-19748
> > > > > > > > >> [4] https://issues.apache.org/jira/browse/FLINK-19717
> > > > > > > > >> [5]
> > > > > > > > >>
> > > > > > > > >>
> > > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://issues.apache.org/jira/issues/?jql=project%20%3D%20FLINK%20AND%20priority%20%3D%20Blocker%20AND%20fixVersion%20%3D%201.11.3
> > > > > > > > >>
> > > > > > > > >
> > > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> >
> > --
> > Best, Jingsong Lee
> >
>

Re: [DISCUSS] Releasing Apache Flink 1.11.3

Posted by Xintong Song <to...@gmail.com>.
Hi folks,

Just to provide some updates concerning the status on the
test instabilities.

Currently, we have 30 unresolved tickets labeled with `Affects Version`
1.11.x.
https://issues.apache.org/jira/browse/FLINK-19775?filter=12348580&jql=project%20%3D%20FLINK%20AND%20resolution%20%3D%20Unresolved%20AND%20affectedVersion%20in%20(1.11.0%2C%201.11.1%2C%201.11.2%2C%201.11.3)%20AND%20labels%20%3D%20test-stability%20ORDER%20BY%20created%20DESC

Among the 30 tickets, 11 of them are:
- Have occured in the recent 3 months
- Not confirmed to be pure testability issues
- Not confirmed to be rare condition cases

It would be nice if someone familiar with these components can take a look
into these issues.

- https://issues.apache.org/jira/browse/FLINK-17159 (ES6)
- https://issues.apache.org/jira/browse/FLINK-17912 (Kafka)
- https://issues.apache.org/jira/browse/FLINK-17949 (Kafka)
⁃ https://issues.apache.org/jira/browse/FLINK-18444 (Kafka)
- https://issues.apache.org/jira/browse/FLINK-18634 (Kafka)
- https://issues.apache.org/jira/browse/FLINK-18648 (Kafka)
- https://issues.apache.org/jira/browse/FLINK-18807 (Kafka)
- https://issues.apache.org/jira/browse/FLINK-19369 (BlobClientTest)
- https://issues.apache.org/jira/browse/FLINK-19436 (TPCDS)
- https://issues.apache.org/jira/browse/FLINK-19690 (Format/Parquet)
- https://issues.apache.org/jira/browse/FLINK-19775
(SystemProcessingTimeServiceTest)

Thank you~

Xintong Song



On Thu, Oct 29, 2020 at 10:21 AM Jingsong Li <ji...@gmail.com> wrote:

> +1 to backport the FLIP-27 adjustments to 1.11.x.
>
> If possible, that would be great. Many people are looking forward to the
> FLIP-27 interface, but they don't want to take the risk to upgrade to 1.12
> (And wait 1.12). After all, 1.11 is a relatively stable version.
>
> Best,
> Jingsong
>
> On Thu, Oct 29, 2020 at 1:24 AM Stephan Ewen <se...@apache.org> wrote:
>
> > Thanks for starting this.
> >
> > +1 form my side to backport the FLIP-27 adjustments to 1.11.x.
> >
> > There were quite a few changes, and I think we need to cherry-pick them
> all
> > to not get some inconsistent mix of changes and many merge conflicts.
> > I made a list below of what we need to add to "release-1.11".
> >
> > *#### Core Source API Changes to backport (in REVERSE order)*
> >
> >   (Use: "git log
> > flink-core/src/main/java/org/apache/flink/api/connector/source")
> >
> > commit 162c072e9265a7b6dd9d6f5459eb7974091c4c4e
> > [FLINK-19492][core] Consolidate Source Events between Source API and
> Split
> > Reader API
> >
> > commit 428fbbad85379f71a9fbf1a1ab3e3dcfa9006436
> > [FLINK-19205][core] Add access to configuration and hostname in the
> > SourceReaderContext
> >
> > commit 723e1790744ddc78d19c7c978442af1383f38d33
> > [hotfix][core] Add to Source Enumerator convenience methods to assign
> > single split
> >
> > *#### Connector Base Changes to Backport (in REVERSE order)*
> >
> >   (Use: "git log flink-connectors/flink-connector-base")
> >
> > commit 401f56fe9d6b0271260edf9787cdcbfe4d03874d
> > [FLINK-19427][FLINK-19489][tests] Fix test conditions for
> > 'SplitFetcherTest.testNotifiesWhenGoingIdleConcurrent()'
> >
> > commit 68c5c2ff779d82a1ff81ffaf60d8a1b283797db1
> > [FLINK-19448][connector base] Explicitly check for un-expected condition
> > that would leave an inconsistent state
> >
> > commit 162c072e9265a7b6dd9d6f5459eb7974091c4c4e
> > [FLINK-19492][core] Consolidate Source Events between Source API and
> Split
> > Reader API
> >
> > commit c1ca7a4c7c21ec8868c14cdddd43c559625b794c
> > [refactor][tests] Move some source test utils from flink-connector-base
> to
> > flink-core
> >
> > commit ee5c4c211c35c70d28252363bbc8400453609977
> > [FLINK-19251][connectors] Avoid confusing queue handling in
> > "SplitReader.handleSplitsChanges()"
> >
> > commit 5abef56b2bf85bcac786f6b16b6899b6cced7176
> > [FLINK-19250][connectors] Fix error propagation in connector base
> > (SplitFetcherManager).
> >
> > commit 8fcca837c55a9216595ee4c03038b52747098dbb
> > [hotfix][connectors] Improve JavaDocs for SingleThreadFetcherManager
> >
> > commit 4700bb5dde3303cbe98882f6beb7379425717b01
> > [FLINK-19225][connectors] Various small improvements to SourceReaderBase
> > (part 2)
> >
> > commit 12261c6b7ed6478a9b9f6a69cb58246b83cab9b7
> > [FLINK-17393][connectors] (follow-up) Wakeup the SplitFetchers more
> > elegantly.
> >
> > commit c60aaff0249bfd6b5871b7f82e03efc487a54d6b
> > [hotfix][tests] Extend test coverage for FutureCompletingBlockingQueue
> >
> > commit cef8a587d7fd2fe64cc644da5ed095d82e46f631
> > [FLINK-19245][connectors] Set default capacity for
> > FutureCompletingBlockingQueue.
> >
> > commit 4ea95782b4c6a2538153d4d16ad3f4839c7de0fb
> > [FLINK-19223][connectors] Simplify Availability Future Model in Base
> > Connector
> >
> > commit 511857049ba30c8ff0ee56da551fa4a479dc583e
> > [FLINK-18128][connectors] Ensure idle split fetchers lead to availability
> > notifications.
> >
> > commit a8206467af0830dcb89623ea068b5ca3b3450c92
> > [refactor][core] Eagerly initialize the FetchTask to support proper unit
> > testing
> >
> > commit 3b2f54bcb437f98e6137c904045cc51072b5c06b
> > [hotfix][tests] Move constants in SplitFetcherTest relevant to only one
> > test into test method
> >
> > commit d7625760a75a508bf05bcddc380bb4d62ee1743e
> > [FLINK-19225][connectors] Various small improvements to SourceReaderBase
> >
> > commit a5b0d3297748c1be47ad579a88f24df2255a8df1
> > [FLINK-17393][connectors] Wakeup the SplitFetchers more elegantly.
> >
> > commit f42a3ebc3e81a034b7221a803c153636fef34903
> > [FLINK-18680][connectors] Make connector base RecordsWithSplitIds more
> > lightweight.
> >
> > commit e3d273de822b085183d09b275a445879ff94b350
> > [FLINK-19162][connectors] Add 'recycle()' to the RecordsWithSplitIds to
> > support reuse of heavy objects.
> >
> > commit 8ebc464c2520453a70001cd712abc8dee6ee89e0
> > [hotfix][testing] Add a set of parameterizable testing mocks for the
> Split
> > Reader API
> >
> > commit 930a07438be1185388d7150640f294dfe2a5d378
> > [hotfix][connectors] Add RequestSplitEvent to 'flink-connector-base'
> >
> > commit e945ce8a933bc378844782f784ca473c767ca159
> > [refactor][connectors] Backport of the connector-base exception handling
> > from the Kafka Connector Pull Request
> >
> >
> > On Wed, Oct 28, 2020 at 5:49 PM Steven Wu <st...@gmail.com> wrote:
> >
> > > I would love to see this FLIP-27 source interface improvement [1] made
> to
> > > 1.11.3.
> > >
> > > [1] https://issues.apache.org/jira/browse/FLINK-19698
> > >
> > > On Wed, Oct 28, 2020 at 12:32 AM Tzu-Li (Gordon) Tai <
> > tzulitai@apache.org>
> > > wrote:
> > >
> > > > Thanks for the replies so far!
> > > >
> > > > Just to provide a brief update on the status of blockers for 1.11.3
> so
> > > far:
> > > >
> > > >
> > > > *PR opened, pending reviewer*- [FLINK-19717]
> SourceReaderBase.pollNext
> > > may
> > > > return END_OF_INPUT if SplitReader.fetch throws (
> > > > https://github.com/apache/flink/pull/13776)
> > > >
> > > > *PR opened, reviewed + close to being merged*
> > > > - [FLINK-19741] Timer service should skip restoring from raw keyed
> > stream
> > > > if it isn't the writer (https://github.com/apache/flink/pull/13761)
> > > > - [FLINK-19748] Raw keyed stream key group iterator should be
> skipping
> > > > unwritten key groups (https://github.com/apache/flink/pull/13772)
> > > >
> > > > *Merged*
> > > > - [FLINK-19154] Application mode deletes HA data in case of suspended
> > > > ZooKeeper connection
> > > > - [FLINK-19569] Upgrade ICU4J to 67.1+
> > > >
> > > > Right now as it seems, progress is mainly blocked on a reviewer for
> > > > FLINK-19717.
> > > > Meanwhile, Xintong is keeping an eye on test instabilities [1] to see
> > if
> > > > there are any fixes that should be applied to `release-1.11`.
> > > >
> > > > This is also a reminder, that if there are other blockers that we
> need
> > to
> > > > be aware of, or a need to re-establish estimated time for getting
> fixes
> > > in
> > > > and delay the RC for 1.11.3, please do let us know!
> > > >
> > > > Cheers,
> > > > Gordon
> > > >
> > > > [1]
> > > >
> > > >
> > >
> >
> https://issues.apache.org/jira/issues/?jql=project%20%3D%20FLINK%20AND%20status%20in%20(Open%2C%20%22In%20Progress%22%2C%20Reopened)%20AND%20affectedVersion%20in%20(1.11.0%2C%201.11.1%2C%201.11.2)%20AND%20labels%20%3D%20test-stability
> > > >
> > > > On Mon, Oct 26, 2020 at 9:43 PM Kostas Kloudas <kk...@apache.org>
> > > > wrote:
> > > >
> > > > > +1 for releasing Flink 1.11.3 as it contains a number of important
> > > > > fixes and thanks Gordon and Xintong for volunteering.
> > > > >
> > > > > Cheers,
> > > > > Kostas
> > > > >
> > > > > On Mon, Oct 26, 2020 at 4:37 AM Yu Li <ca...@gmail.com> wrote:
> > > > > >
> > > > > > +1 for releasing Flink 1.11.3, and thanks Gordon and Xintong for
> > > > > > volunteering as our release managers.
> > > > > >
> > > > > > Best Regards,
> > > > > > Yu
> > > > > >
> > > > > >
> > > > > > On Mon, 26 Oct 2020 at 09:45, Xintong Song <
> tonysong820@gmail.com>
> > > > > wrote:
> > > > > >
> > > > > > > Thanks Gordan for starting this discussion.
> > > > > > > My pleasure to help with the release process.
> > > > > > >
> > > > > > > Thank you~
> > > > > > >
> > > > > > > Xintong Song
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On Fri, Oct 23, 2020 at 11:29 PM Till Rohrmann <
> > > trohrmann@apache.org
> > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Thanks for starting this discussion Gordon. There are over
> 100
> > > > issues
> > > > > > > > which are fixed for 1.11.3. Hence +1 for a soonish 1.11.3
> > > release.
> > > > > Thanks
> > > > > > > > for volunteering as our release managers Gordon and Xintong!
> > > > > > > >
> > > > > > > > Cheers,
> > > > > > > > Till
> > > > > > > >
> > > > > > > > On Fri, Oct 23, 2020 at 5:02 PM Tzu-Li (Gordon) Tai <
> > > > > tzulitai@apache.org
> > > > > > > >
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > >> Hi,
> > > > > > > >>
> > > > > > > >> Xintong and I would like to start a discussion for releasing
> > > Flink
> > > > > > > 1.11.3
> > > > > > > >> soon.
> > > > > > > >>
> > > > > > > >> It seems like we already have a few pressing issues that
> needs
> > > to
> > > > be
> > > > > > > >> included in a new hotfix release:
> > > > > > > >>
> > > > > > > >>    - Heap-based timers’ restore behaviour is causing a
> > critical
> > > > > recovery
> > > > > > > >>    issue for StateFun [1] [2] [3].
> > > > > > > >>    - There are several robustness issues for the FLIP-27 new
> > > > source
> > > > > API,
> > > > > > > >>    such as [4]. We already have some users using the FLIP-27
> > API
> > > > > with
> > > > > > > >> 1.11.x,
> > > > > > > >>    so it would be important to get those fixes in for 1.11.x
> > as
> > > > > well.
> > > > > > > >>
> > > > > > > >> Apart from the issues that are already marked as blocker for
> > > > 1.11.3
> > > > > in
> > > > > > > our
> > > > > > > >> JIRA [5], please let us know in this thread if there is
> > already
> > > > > ongoing
> > > > > > > >> work for other important fixes that we should try to
> include.
> > > > > > > >>
> > > > > > > >> Xintong and I would like to volunteer for managing this
> > release,
> > > > and
> > > > > > > will
> > > > > > > >> try to communicate the priority of pending blockers over the
> > > next
> > > > > few
> > > > > > > >> days.
> > > > > > > >> Since the aforementioned issues are quite critical, we’d
> like
> > to
> > > > aim
> > > > > > > >> for a *feature
> > > > > > > >> freeze by the end of next week (Oct. 30th)* and start the
> > > release
> > > > > voting
> > > > > > > >> process the week after.
> > > > > > > >> If that is too short of a notice and you might need more
> time,
> > > > > please
> > > > > > > let
> > > > > > > >> us know!
> > > > > > > >>
> > > > > > > >> Cheers,
> > > > > > > >> Gordon
> > > > > > > >>
> > > > > > > >> [1] https://issues.apache.org/jira/browse/FLINK-19692
> > > > > > > >> [2] https://issues.apache.org/jira/browse/FLINK-19741
> > > > > > > >> [3] https://issues.apache.org/jira/browse/FLINK-19748
> > > > > > > >> [4] https://issues.apache.org/jira/browse/FLINK-19717
> > > > > > > >> [5]
> > > > > > > >>
> > > > > > > >>
> > > > > > >
> > > > >
> > > >
> > >
> >
> https://issues.apache.org/jira/issues/?jql=project%20%3D%20FLINK%20AND%20priority%20%3D%20Blocker%20AND%20fixVersion%20%3D%201.11.3
> > > > > > > >>
> > > > > > > >
> > > > > > >
> > > > >
> > > >
> > >
> >
>
>
> --
> Best, Jingsong Lee
>

Re: [DISCUSS] Releasing Apache Flink 1.11.3

Posted by Jingsong Li <ji...@gmail.com>.
+1 to backport the FLIP-27 adjustments to 1.11.x.

If possible, that would be great. Many people are looking forward to the
FLIP-27 interface, but they don't want to take the risk to upgrade to 1.12
(And wait 1.12). After all, 1.11 is a relatively stable version.

Best,
Jingsong

On Thu, Oct 29, 2020 at 1:24 AM Stephan Ewen <se...@apache.org> wrote:

> Thanks for starting this.
>
> +1 form my side to backport the FLIP-27 adjustments to 1.11.x.
>
> There were quite a few changes, and I think we need to cherry-pick them all
> to not get some inconsistent mix of changes and many merge conflicts.
> I made a list below of what we need to add to "release-1.11".
>
> *#### Core Source API Changes to backport (in REVERSE order)*
>
>   (Use: "git log
> flink-core/src/main/java/org/apache/flink/api/connector/source")
>
> commit 162c072e9265a7b6dd9d6f5459eb7974091c4c4e
> [FLINK-19492][core] Consolidate Source Events between Source API and Split
> Reader API
>
> commit 428fbbad85379f71a9fbf1a1ab3e3dcfa9006436
> [FLINK-19205][core] Add access to configuration and hostname in the
> SourceReaderContext
>
> commit 723e1790744ddc78d19c7c978442af1383f38d33
> [hotfix][core] Add to Source Enumerator convenience methods to assign
> single split
>
> *#### Connector Base Changes to Backport (in REVERSE order)*
>
>   (Use: "git log flink-connectors/flink-connector-base")
>
> commit 401f56fe9d6b0271260edf9787cdcbfe4d03874d
> [FLINK-19427][FLINK-19489][tests] Fix test conditions for
> 'SplitFetcherTest.testNotifiesWhenGoingIdleConcurrent()'
>
> commit 68c5c2ff779d82a1ff81ffaf60d8a1b283797db1
> [FLINK-19448][connector base] Explicitly check for un-expected condition
> that would leave an inconsistent state
>
> commit 162c072e9265a7b6dd9d6f5459eb7974091c4c4e
> [FLINK-19492][core] Consolidate Source Events between Source API and Split
> Reader API
>
> commit c1ca7a4c7c21ec8868c14cdddd43c559625b794c
> [refactor][tests] Move some source test utils from flink-connector-base to
> flink-core
>
> commit ee5c4c211c35c70d28252363bbc8400453609977
> [FLINK-19251][connectors] Avoid confusing queue handling in
> "SplitReader.handleSplitsChanges()"
>
> commit 5abef56b2bf85bcac786f6b16b6899b6cced7176
> [FLINK-19250][connectors] Fix error propagation in connector base
> (SplitFetcherManager).
>
> commit 8fcca837c55a9216595ee4c03038b52747098dbb
> [hotfix][connectors] Improve JavaDocs for SingleThreadFetcherManager
>
> commit 4700bb5dde3303cbe98882f6beb7379425717b01
> [FLINK-19225][connectors] Various small improvements to SourceReaderBase
> (part 2)
>
> commit 12261c6b7ed6478a9b9f6a69cb58246b83cab9b7
> [FLINK-17393][connectors] (follow-up) Wakeup the SplitFetchers more
> elegantly.
>
> commit c60aaff0249bfd6b5871b7f82e03efc487a54d6b
> [hotfix][tests] Extend test coverage for FutureCompletingBlockingQueue
>
> commit cef8a587d7fd2fe64cc644da5ed095d82e46f631
> [FLINK-19245][connectors] Set default capacity for
> FutureCompletingBlockingQueue.
>
> commit 4ea95782b4c6a2538153d4d16ad3f4839c7de0fb
> [FLINK-19223][connectors] Simplify Availability Future Model in Base
> Connector
>
> commit 511857049ba30c8ff0ee56da551fa4a479dc583e
> [FLINK-18128][connectors] Ensure idle split fetchers lead to availability
> notifications.
>
> commit a8206467af0830dcb89623ea068b5ca3b3450c92
> [refactor][core] Eagerly initialize the FetchTask to support proper unit
> testing
>
> commit 3b2f54bcb437f98e6137c904045cc51072b5c06b
> [hotfix][tests] Move constants in SplitFetcherTest relevant to only one
> test into test method
>
> commit d7625760a75a508bf05bcddc380bb4d62ee1743e
> [FLINK-19225][connectors] Various small improvements to SourceReaderBase
>
> commit a5b0d3297748c1be47ad579a88f24df2255a8df1
> [FLINK-17393][connectors] Wakeup the SplitFetchers more elegantly.
>
> commit f42a3ebc3e81a034b7221a803c153636fef34903
> [FLINK-18680][connectors] Make connector base RecordsWithSplitIds more
> lightweight.
>
> commit e3d273de822b085183d09b275a445879ff94b350
> [FLINK-19162][connectors] Add 'recycle()' to the RecordsWithSplitIds to
> support reuse of heavy objects.
>
> commit 8ebc464c2520453a70001cd712abc8dee6ee89e0
> [hotfix][testing] Add a set of parameterizable testing mocks for the Split
> Reader API
>
> commit 930a07438be1185388d7150640f294dfe2a5d378
> [hotfix][connectors] Add RequestSplitEvent to 'flink-connector-base'
>
> commit e945ce8a933bc378844782f784ca473c767ca159
> [refactor][connectors] Backport of the connector-base exception handling
> from the Kafka Connector Pull Request
>
>
> On Wed, Oct 28, 2020 at 5:49 PM Steven Wu <st...@gmail.com> wrote:
>
> > I would love to see this FLIP-27 source interface improvement [1] made to
> > 1.11.3.
> >
> > [1] https://issues.apache.org/jira/browse/FLINK-19698
> >
> > On Wed, Oct 28, 2020 at 12:32 AM Tzu-Li (Gordon) Tai <
> tzulitai@apache.org>
> > wrote:
> >
> > > Thanks for the replies so far!
> > >
> > > Just to provide a brief update on the status of blockers for 1.11.3 so
> > far:
> > >
> > >
> > > *PR opened, pending reviewer*- [FLINK-19717] SourceReaderBase.pollNext
> > may
> > > return END_OF_INPUT if SplitReader.fetch throws (
> > > https://github.com/apache/flink/pull/13776)
> > >
> > > *PR opened, reviewed + close to being merged*
> > > - [FLINK-19741] Timer service should skip restoring from raw keyed
> stream
> > > if it isn't the writer (https://github.com/apache/flink/pull/13761)
> > > - [FLINK-19748] Raw keyed stream key group iterator should be skipping
> > > unwritten key groups (https://github.com/apache/flink/pull/13772)
> > >
> > > *Merged*
> > > - [FLINK-19154] Application mode deletes HA data in case of suspended
> > > ZooKeeper connection
> > > - [FLINK-19569] Upgrade ICU4J to 67.1+
> > >
> > > Right now as it seems, progress is mainly blocked on a reviewer for
> > > FLINK-19717.
> > > Meanwhile, Xintong is keeping an eye on test instabilities [1] to see
> if
> > > there are any fixes that should be applied to `release-1.11`.
> > >
> > > This is also a reminder, that if there are other blockers that we need
> to
> > > be aware of, or a need to re-establish estimated time for getting fixes
> > in
> > > and delay the RC for 1.11.3, please do let us know!
> > >
> > > Cheers,
> > > Gordon
> > >
> > > [1]
> > >
> > >
> >
> https://issues.apache.org/jira/issues/?jql=project%20%3D%20FLINK%20AND%20status%20in%20(Open%2C%20%22In%20Progress%22%2C%20Reopened)%20AND%20affectedVersion%20in%20(1.11.0%2C%201.11.1%2C%201.11.2)%20AND%20labels%20%3D%20test-stability
> > >
> > > On Mon, Oct 26, 2020 at 9:43 PM Kostas Kloudas <kk...@apache.org>
> > > wrote:
> > >
> > > > +1 for releasing Flink 1.11.3 as it contains a number of important
> > > > fixes and thanks Gordon and Xintong for volunteering.
> > > >
> > > > Cheers,
> > > > Kostas
> > > >
> > > > On Mon, Oct 26, 2020 at 4:37 AM Yu Li <ca...@gmail.com> wrote:
> > > > >
> > > > > +1 for releasing Flink 1.11.3, and thanks Gordon and Xintong for
> > > > > volunteering as our release managers.
> > > > >
> > > > > Best Regards,
> > > > > Yu
> > > > >
> > > > >
> > > > > On Mon, 26 Oct 2020 at 09:45, Xintong Song <to...@gmail.com>
> > > > wrote:
> > > > >
> > > > > > Thanks Gordan for starting this discussion.
> > > > > > My pleasure to help with the release process.
> > > > > >
> > > > > > Thank you~
> > > > > >
> > > > > > Xintong Song
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Fri, Oct 23, 2020 at 11:29 PM Till Rohrmann <
> > trohrmann@apache.org
> > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Thanks for starting this discussion Gordon. There are over 100
> > > issues
> > > > > > > which are fixed for 1.11.3. Hence +1 for a soonish 1.11.3
> > release.
> > > > Thanks
> > > > > > > for volunteering as our release managers Gordon and Xintong!
> > > > > > >
> > > > > > > Cheers,
> > > > > > > Till
> > > > > > >
> > > > > > > On Fri, Oct 23, 2020 at 5:02 PM Tzu-Li (Gordon) Tai <
> > > > tzulitai@apache.org
> > > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > >> Hi,
> > > > > > >>
> > > > > > >> Xintong and I would like to start a discussion for releasing
> > Flink
> > > > > > 1.11.3
> > > > > > >> soon.
> > > > > > >>
> > > > > > >> It seems like we already have a few pressing issues that needs
> > to
> > > be
> > > > > > >> included in a new hotfix release:
> > > > > > >>
> > > > > > >>    - Heap-based timers’ restore behaviour is causing a
> critical
> > > > recovery
> > > > > > >>    issue for StateFun [1] [2] [3].
> > > > > > >>    - There are several robustness issues for the FLIP-27 new
> > > source
> > > > API,
> > > > > > >>    such as [4]. We already have some users using the FLIP-27
> API
> > > > with
> > > > > > >> 1.11.x,
> > > > > > >>    so it would be important to get those fixes in for 1.11.x
> as
> > > > well.
> > > > > > >>
> > > > > > >> Apart from the issues that are already marked as blocker for
> > > 1.11.3
> > > > in
> > > > > > our
> > > > > > >> JIRA [5], please let us know in this thread if there is
> already
> > > > ongoing
> > > > > > >> work for other important fixes that we should try to include.
> > > > > > >>
> > > > > > >> Xintong and I would like to volunteer for managing this
> release,
> > > and
> > > > > > will
> > > > > > >> try to communicate the priority of pending blockers over the
> > next
> > > > few
> > > > > > >> days.
> > > > > > >> Since the aforementioned issues are quite critical, we’d like
> to
> > > aim
> > > > > > >> for a *feature
> > > > > > >> freeze by the end of next week (Oct. 30th)* and start the
> > release
> > > > voting
> > > > > > >> process the week after.
> > > > > > >> If that is too short of a notice and you might need more time,
> > > > please
> > > > > > let
> > > > > > >> us know!
> > > > > > >>
> > > > > > >> Cheers,
> > > > > > >> Gordon
> > > > > > >>
> > > > > > >> [1] https://issues.apache.org/jira/browse/FLINK-19692
> > > > > > >> [2] https://issues.apache.org/jira/browse/FLINK-19741
> > > > > > >> [3] https://issues.apache.org/jira/browse/FLINK-19748
> > > > > > >> [4] https://issues.apache.org/jira/browse/FLINK-19717
> > > > > > >> [5]
> > > > > > >>
> > > > > > >>
> > > > > >
> > > >
> > >
> >
> https://issues.apache.org/jira/issues/?jql=project%20%3D%20FLINK%20AND%20priority%20%3D%20Blocker%20AND%20fixVersion%20%3D%201.11.3
> > > > > > >>
> > > > > > >
> > > > > >
> > > >
> > >
> >
>


-- 
Best, Jingsong Lee

Re: [DISCUSS] Releasing Apache Flink 1.11.3

Posted by Stephan Ewen <se...@apache.org>.
Thanks for starting this.

+1 form my side to backport the FLIP-27 adjustments to 1.11.x.

There were quite a few changes, and I think we need to cherry-pick them all
to not get some inconsistent mix of changes and many merge conflicts.
I made a list below of what we need to add to "release-1.11".

*#### Core Source API Changes to backport (in REVERSE order)*

  (Use: "git log
flink-core/src/main/java/org/apache/flink/api/connector/source")

commit 162c072e9265a7b6dd9d6f5459eb7974091c4c4e
[FLINK-19492][core] Consolidate Source Events between Source API and Split
Reader API

commit 428fbbad85379f71a9fbf1a1ab3e3dcfa9006436
[FLINK-19205][core] Add access to configuration and hostname in the
SourceReaderContext

commit 723e1790744ddc78d19c7c978442af1383f38d33
[hotfix][core] Add to Source Enumerator convenience methods to assign
single split

*#### Connector Base Changes to Backport (in REVERSE order)*

  (Use: "git log flink-connectors/flink-connector-base")

commit 401f56fe9d6b0271260edf9787cdcbfe4d03874d
[FLINK-19427][FLINK-19489][tests] Fix test conditions for
'SplitFetcherTest.testNotifiesWhenGoingIdleConcurrent()'

commit 68c5c2ff779d82a1ff81ffaf60d8a1b283797db1
[FLINK-19448][connector base] Explicitly check for un-expected condition
that would leave an inconsistent state

commit 162c072e9265a7b6dd9d6f5459eb7974091c4c4e
[FLINK-19492][core] Consolidate Source Events between Source API and Split
Reader API

commit c1ca7a4c7c21ec8868c14cdddd43c559625b794c
[refactor][tests] Move some source test utils from flink-connector-base to
flink-core

commit ee5c4c211c35c70d28252363bbc8400453609977
[FLINK-19251][connectors] Avoid confusing queue handling in
"SplitReader.handleSplitsChanges()"

commit 5abef56b2bf85bcac786f6b16b6899b6cced7176
[FLINK-19250][connectors] Fix error propagation in connector base
(SplitFetcherManager).

commit 8fcca837c55a9216595ee4c03038b52747098dbb
[hotfix][connectors] Improve JavaDocs for SingleThreadFetcherManager

commit 4700bb5dde3303cbe98882f6beb7379425717b01
[FLINK-19225][connectors] Various small improvements to SourceReaderBase
(part 2)

commit 12261c6b7ed6478a9b9f6a69cb58246b83cab9b7
[FLINK-17393][connectors] (follow-up) Wakeup the SplitFetchers more
elegantly.

commit c60aaff0249bfd6b5871b7f82e03efc487a54d6b
[hotfix][tests] Extend test coverage for FutureCompletingBlockingQueue

commit cef8a587d7fd2fe64cc644da5ed095d82e46f631
[FLINK-19245][connectors] Set default capacity for
FutureCompletingBlockingQueue.

commit 4ea95782b4c6a2538153d4d16ad3f4839c7de0fb
[FLINK-19223][connectors] Simplify Availability Future Model in Base
Connector

commit 511857049ba30c8ff0ee56da551fa4a479dc583e
[FLINK-18128][connectors] Ensure idle split fetchers lead to availability
notifications.

commit a8206467af0830dcb89623ea068b5ca3b3450c92
[refactor][core] Eagerly initialize the FetchTask to support proper unit
testing

commit 3b2f54bcb437f98e6137c904045cc51072b5c06b
[hotfix][tests] Move constants in SplitFetcherTest relevant to only one
test into test method

commit d7625760a75a508bf05bcddc380bb4d62ee1743e
[FLINK-19225][connectors] Various small improvements to SourceReaderBase

commit a5b0d3297748c1be47ad579a88f24df2255a8df1
[FLINK-17393][connectors] Wakeup the SplitFetchers more elegantly.

commit f42a3ebc3e81a034b7221a803c153636fef34903
[FLINK-18680][connectors] Make connector base RecordsWithSplitIds more
lightweight.

commit e3d273de822b085183d09b275a445879ff94b350
[FLINK-19162][connectors] Add 'recycle()' to the RecordsWithSplitIds to
support reuse of heavy objects.

commit 8ebc464c2520453a70001cd712abc8dee6ee89e0
[hotfix][testing] Add a set of parameterizable testing mocks for the Split
Reader API

commit 930a07438be1185388d7150640f294dfe2a5d378
[hotfix][connectors] Add RequestSplitEvent to 'flink-connector-base'

commit e945ce8a933bc378844782f784ca473c767ca159
[refactor][connectors] Backport of the connector-base exception handling
from the Kafka Connector Pull Request


On Wed, Oct 28, 2020 at 5:49 PM Steven Wu <st...@gmail.com> wrote:

> I would love to see this FLIP-27 source interface improvement [1] made to
> 1.11.3.
>
> [1] https://issues.apache.org/jira/browse/FLINK-19698
>
> On Wed, Oct 28, 2020 at 12:32 AM Tzu-Li (Gordon) Tai <tz...@apache.org>
> wrote:
>
> > Thanks for the replies so far!
> >
> > Just to provide a brief update on the status of blockers for 1.11.3 so
> far:
> >
> >
> > *PR opened, pending reviewer*- [FLINK-19717] SourceReaderBase.pollNext
> may
> > return END_OF_INPUT if SplitReader.fetch throws (
> > https://github.com/apache/flink/pull/13776)
> >
> > *PR opened, reviewed + close to being merged*
> > - [FLINK-19741] Timer service should skip restoring from raw keyed stream
> > if it isn't the writer (https://github.com/apache/flink/pull/13761)
> > - [FLINK-19748] Raw keyed stream key group iterator should be skipping
> > unwritten key groups (https://github.com/apache/flink/pull/13772)
> >
> > *Merged*
> > - [FLINK-19154] Application mode deletes HA data in case of suspended
> > ZooKeeper connection
> > - [FLINK-19569] Upgrade ICU4J to 67.1+
> >
> > Right now as it seems, progress is mainly blocked on a reviewer for
> > FLINK-19717.
> > Meanwhile, Xintong is keeping an eye on test instabilities [1] to see if
> > there are any fixes that should be applied to `release-1.11`.
> >
> > This is also a reminder, that if there are other blockers that we need to
> > be aware of, or a need to re-establish estimated time for getting fixes
> in
> > and delay the RC for 1.11.3, please do let us know!
> >
> > Cheers,
> > Gordon
> >
> > [1]
> >
> >
> https://issues.apache.org/jira/issues/?jql=project%20%3D%20FLINK%20AND%20status%20in%20(Open%2C%20%22In%20Progress%22%2C%20Reopened)%20AND%20affectedVersion%20in%20(1.11.0%2C%201.11.1%2C%201.11.2)%20AND%20labels%20%3D%20test-stability
> >
> > On Mon, Oct 26, 2020 at 9:43 PM Kostas Kloudas <kk...@apache.org>
> > wrote:
> >
> > > +1 for releasing Flink 1.11.3 as it contains a number of important
> > > fixes and thanks Gordon and Xintong for volunteering.
> > >
> > > Cheers,
> > > Kostas
> > >
> > > On Mon, Oct 26, 2020 at 4:37 AM Yu Li <ca...@gmail.com> wrote:
> > > >
> > > > +1 for releasing Flink 1.11.3, and thanks Gordon and Xintong for
> > > > volunteering as our release managers.
> > > >
> > > > Best Regards,
> > > > Yu
> > > >
> > > >
> > > > On Mon, 26 Oct 2020 at 09:45, Xintong Song <to...@gmail.com>
> > > wrote:
> > > >
> > > > > Thanks Gordan for starting this discussion.
> > > > > My pleasure to help with the release process.
> > > > >
> > > > > Thank you~
> > > > >
> > > > > Xintong Song
> > > > >
> > > > >
> > > > >
> > > > > On Fri, Oct 23, 2020 at 11:29 PM Till Rohrmann <
> trohrmann@apache.org
> > >
> > > > > wrote:
> > > > >
> > > > > > Thanks for starting this discussion Gordon. There are over 100
> > issues
> > > > > > which are fixed for 1.11.3. Hence +1 for a soonish 1.11.3
> release.
> > > Thanks
> > > > > > for volunteering as our release managers Gordon and Xintong!
> > > > > >
> > > > > > Cheers,
> > > > > > Till
> > > > > >
> > > > > > On Fri, Oct 23, 2020 at 5:02 PM Tzu-Li (Gordon) Tai <
> > > tzulitai@apache.org
> > > > > >
> > > > > > wrote:
> > > > > >
> > > > > >> Hi,
> > > > > >>
> > > > > >> Xintong and I would like to start a discussion for releasing
> Flink
> > > > > 1.11.3
> > > > > >> soon.
> > > > > >>
> > > > > >> It seems like we already have a few pressing issues that needs
> to
> > be
> > > > > >> included in a new hotfix release:
> > > > > >>
> > > > > >>    - Heap-based timers’ restore behaviour is causing a critical
> > > recovery
> > > > > >>    issue for StateFun [1] [2] [3].
> > > > > >>    - There are several robustness issues for the FLIP-27 new
> > source
> > > API,
> > > > > >>    such as [4]. We already have some users using the FLIP-27 API
> > > with
> > > > > >> 1.11.x,
> > > > > >>    so it would be important to get those fixes in for 1.11.x as
> > > well.
> > > > > >>
> > > > > >> Apart from the issues that are already marked as blocker for
> > 1.11.3
> > > in
> > > > > our
> > > > > >> JIRA [5], please let us know in this thread if there is already
> > > ongoing
> > > > > >> work for other important fixes that we should try to include.
> > > > > >>
> > > > > >> Xintong and I would like to volunteer for managing this release,
> > and
> > > > > will
> > > > > >> try to communicate the priority of pending blockers over the
> next
> > > few
> > > > > >> days.
> > > > > >> Since the aforementioned issues are quite critical, we’d like to
> > aim
> > > > > >> for a *feature
> > > > > >> freeze by the end of next week (Oct. 30th)* and start the
> release
> > > voting
> > > > > >> process the week after.
> > > > > >> If that is too short of a notice and you might need more time,
> > > please
> > > > > let
> > > > > >> us know!
> > > > > >>
> > > > > >> Cheers,
> > > > > >> Gordon
> > > > > >>
> > > > > >> [1] https://issues.apache.org/jira/browse/FLINK-19692
> > > > > >> [2] https://issues.apache.org/jira/browse/FLINK-19741
> > > > > >> [3] https://issues.apache.org/jira/browse/FLINK-19748
> > > > > >> [4] https://issues.apache.org/jira/browse/FLINK-19717
> > > > > >> [5]
> > > > > >>
> > > > > >>
> > > > >
> > >
> >
> https://issues.apache.org/jira/issues/?jql=project%20%3D%20FLINK%20AND%20priority%20%3D%20Blocker%20AND%20fixVersion%20%3D%201.11.3
> > > > > >>
> > > > > >
> > > > >
> > >
> >
>

Re: [DISCUSS] Releasing Apache Flink 1.11.3

Posted by Steven Wu <st...@gmail.com>.
I would love to see this FLIP-27 source interface improvement [1] made to
1.11.3.

[1] https://issues.apache.org/jira/browse/FLINK-19698

On Wed, Oct 28, 2020 at 12:32 AM Tzu-Li (Gordon) Tai <tz...@apache.org>
wrote:

> Thanks for the replies so far!
>
> Just to provide a brief update on the status of blockers for 1.11.3 so far:
>
>
> *PR opened, pending reviewer*- [FLINK-19717] SourceReaderBase.pollNext may
> return END_OF_INPUT if SplitReader.fetch throws (
> https://github.com/apache/flink/pull/13776)
>
> *PR opened, reviewed + close to being merged*
> - [FLINK-19741] Timer service should skip restoring from raw keyed stream
> if it isn't the writer (https://github.com/apache/flink/pull/13761)
> - [FLINK-19748] Raw keyed stream key group iterator should be skipping
> unwritten key groups (https://github.com/apache/flink/pull/13772)
>
> *Merged*
> - [FLINK-19154] Application mode deletes HA data in case of suspended
> ZooKeeper connection
> - [FLINK-19569] Upgrade ICU4J to 67.1+
>
> Right now as it seems, progress is mainly blocked on a reviewer for
> FLINK-19717.
> Meanwhile, Xintong is keeping an eye on test instabilities [1] to see if
> there are any fixes that should be applied to `release-1.11`.
>
> This is also a reminder, that if there are other blockers that we need to
> be aware of, or a need to re-establish estimated time for getting fixes in
> and delay the RC for 1.11.3, please do let us know!
>
> Cheers,
> Gordon
>
> [1]
>
> https://issues.apache.org/jira/issues/?jql=project%20%3D%20FLINK%20AND%20status%20in%20(Open%2C%20%22In%20Progress%22%2C%20Reopened)%20AND%20affectedVersion%20in%20(1.11.0%2C%201.11.1%2C%201.11.2)%20AND%20labels%20%3D%20test-stability
>
> On Mon, Oct 26, 2020 at 9:43 PM Kostas Kloudas <kk...@apache.org>
> wrote:
>
> > +1 for releasing Flink 1.11.3 as it contains a number of important
> > fixes and thanks Gordon and Xintong for volunteering.
> >
> > Cheers,
> > Kostas
> >
> > On Mon, Oct 26, 2020 at 4:37 AM Yu Li <ca...@gmail.com> wrote:
> > >
> > > +1 for releasing Flink 1.11.3, and thanks Gordon and Xintong for
> > > volunteering as our release managers.
> > >
> > > Best Regards,
> > > Yu
> > >
> > >
> > > On Mon, 26 Oct 2020 at 09:45, Xintong Song <to...@gmail.com>
> > wrote:
> > >
> > > > Thanks Gordan for starting this discussion.
> > > > My pleasure to help with the release process.
> > > >
> > > > Thank you~
> > > >
> > > > Xintong Song
> > > >
> > > >
> > > >
> > > > On Fri, Oct 23, 2020 at 11:29 PM Till Rohrmann <trohrmann@apache.org
> >
> > > > wrote:
> > > >
> > > > > Thanks for starting this discussion Gordon. There are over 100
> issues
> > > > > which are fixed for 1.11.3. Hence +1 for a soonish 1.11.3 release.
> > Thanks
> > > > > for volunteering as our release managers Gordon and Xintong!
> > > > >
> > > > > Cheers,
> > > > > Till
> > > > >
> > > > > On Fri, Oct 23, 2020 at 5:02 PM Tzu-Li (Gordon) Tai <
> > tzulitai@apache.org
> > > > >
> > > > > wrote:
> > > > >
> > > > >> Hi,
> > > > >>
> > > > >> Xintong and I would like to start a discussion for releasing Flink
> > > > 1.11.3
> > > > >> soon.
> > > > >>
> > > > >> It seems like we already have a few pressing issues that needs to
> be
> > > > >> included in a new hotfix release:
> > > > >>
> > > > >>    - Heap-based timers’ restore behaviour is causing a critical
> > recovery
> > > > >>    issue for StateFun [1] [2] [3].
> > > > >>    - There are several robustness issues for the FLIP-27 new
> source
> > API,
> > > > >>    such as [4]. We already have some users using the FLIP-27 API
> > with
> > > > >> 1.11.x,
> > > > >>    so it would be important to get those fixes in for 1.11.x as
> > well.
> > > > >>
> > > > >> Apart from the issues that are already marked as blocker for
> 1.11.3
> > in
> > > > our
> > > > >> JIRA [5], please let us know in this thread if there is already
> > ongoing
> > > > >> work for other important fixes that we should try to include.
> > > > >>
> > > > >> Xintong and I would like to volunteer for managing this release,
> and
> > > > will
> > > > >> try to communicate the priority of pending blockers over the next
> > few
> > > > >> days.
> > > > >> Since the aforementioned issues are quite critical, we’d like to
> aim
> > > > >> for a *feature
> > > > >> freeze by the end of next week (Oct. 30th)* and start the release
> > voting
> > > > >> process the week after.
> > > > >> If that is too short of a notice and you might need more time,
> > please
> > > > let
> > > > >> us know!
> > > > >>
> > > > >> Cheers,
> > > > >> Gordon
> > > > >>
> > > > >> [1] https://issues.apache.org/jira/browse/FLINK-19692
> > > > >> [2] https://issues.apache.org/jira/browse/FLINK-19741
> > > > >> [3] https://issues.apache.org/jira/browse/FLINK-19748
> > > > >> [4] https://issues.apache.org/jira/browse/FLINK-19717
> > > > >> [5]
> > > > >>
> > > > >>
> > > >
> >
> https://issues.apache.org/jira/issues/?jql=project%20%3D%20FLINK%20AND%20priority%20%3D%20Blocker%20AND%20fixVersion%20%3D%201.11.3
> > > > >>
> > > > >
> > > >
> >
>

Re: [DISCUSS] Releasing Apache Flink 1.11.3

Posted by "Tzu-Li (Gordon) Tai" <tz...@apache.org>.
Thanks for the replies so far!

Just to provide a brief update on the status of blockers for 1.11.3 so far:


*PR opened, pending reviewer*- [FLINK-19717] SourceReaderBase.pollNext may
return END_OF_INPUT if SplitReader.fetch throws (
https://github.com/apache/flink/pull/13776)

*PR opened, reviewed + close to being merged*
- [FLINK-19741] Timer service should skip restoring from raw keyed stream
if it isn't the writer (https://github.com/apache/flink/pull/13761)
- [FLINK-19748] Raw keyed stream key group iterator should be skipping
unwritten key groups (https://github.com/apache/flink/pull/13772)

*Merged*
- [FLINK-19154] Application mode deletes HA data in case of suspended
ZooKeeper connection
- [FLINK-19569] Upgrade ICU4J to 67.1+

Right now as it seems, progress is mainly blocked on a reviewer for
FLINK-19717.
Meanwhile, Xintong is keeping an eye on test instabilities [1] to see if
there are any fixes that should be applied to `release-1.11`.

This is also a reminder, that if there are other blockers that we need to
be aware of, or a need to re-establish estimated time for getting fixes in
and delay the RC for 1.11.3, please do let us know!

Cheers,
Gordon

[1]
https://issues.apache.org/jira/issues/?jql=project%20%3D%20FLINK%20AND%20status%20in%20(Open%2C%20%22In%20Progress%22%2C%20Reopened)%20AND%20affectedVersion%20in%20(1.11.0%2C%201.11.1%2C%201.11.2)%20AND%20labels%20%3D%20test-stability

On Mon, Oct 26, 2020 at 9:43 PM Kostas Kloudas <kk...@apache.org> wrote:

> +1 for releasing Flink 1.11.3 as it contains a number of important
> fixes and thanks Gordon and Xintong for volunteering.
>
> Cheers,
> Kostas
>
> On Mon, Oct 26, 2020 at 4:37 AM Yu Li <ca...@gmail.com> wrote:
> >
> > +1 for releasing Flink 1.11.3, and thanks Gordon and Xintong for
> > volunteering as our release managers.
> >
> > Best Regards,
> > Yu
> >
> >
> > On Mon, 26 Oct 2020 at 09:45, Xintong Song <to...@gmail.com>
> wrote:
> >
> > > Thanks Gordan for starting this discussion.
> > > My pleasure to help with the release process.
> > >
> > > Thank you~
> > >
> > > Xintong Song
> > >
> > >
> > >
> > > On Fri, Oct 23, 2020 at 11:29 PM Till Rohrmann <tr...@apache.org>
> > > wrote:
> > >
> > > > Thanks for starting this discussion Gordon. There are over 100 issues
> > > > which are fixed for 1.11.3. Hence +1 for a soonish 1.11.3 release.
> Thanks
> > > > for volunteering as our release managers Gordon and Xintong!
> > > >
> > > > Cheers,
> > > > Till
> > > >
> > > > On Fri, Oct 23, 2020 at 5:02 PM Tzu-Li (Gordon) Tai <
> tzulitai@apache.org
> > > >
> > > > wrote:
> > > >
> > > >> Hi,
> > > >>
> > > >> Xintong and I would like to start a discussion for releasing Flink
> > > 1.11.3
> > > >> soon.
> > > >>
> > > >> It seems like we already have a few pressing issues that needs to be
> > > >> included in a new hotfix release:
> > > >>
> > > >>    - Heap-based timers’ restore behaviour is causing a critical
> recovery
> > > >>    issue for StateFun [1] [2] [3].
> > > >>    - There are several robustness issues for the FLIP-27 new source
> API,
> > > >>    such as [4]. We already have some users using the FLIP-27 API
> with
> > > >> 1.11.x,
> > > >>    so it would be important to get those fixes in for 1.11.x as
> well.
> > > >>
> > > >> Apart from the issues that are already marked as blocker for 1.11.3
> in
> > > our
> > > >> JIRA [5], please let us know in this thread if there is already
> ongoing
> > > >> work for other important fixes that we should try to include.
> > > >>
> > > >> Xintong and I would like to volunteer for managing this release, and
> > > will
> > > >> try to communicate the priority of pending blockers over the next
> few
> > > >> days.
> > > >> Since the aforementioned issues are quite critical, we’d like to aim
> > > >> for a *feature
> > > >> freeze by the end of next week (Oct. 30th)* and start the release
> voting
> > > >> process the week after.
> > > >> If that is too short of a notice and you might need more time,
> please
> > > let
> > > >> us know!
> > > >>
> > > >> Cheers,
> > > >> Gordon
> > > >>
> > > >> [1] https://issues.apache.org/jira/browse/FLINK-19692
> > > >> [2] https://issues.apache.org/jira/browse/FLINK-19741
> > > >> [3] https://issues.apache.org/jira/browse/FLINK-19748
> > > >> [4] https://issues.apache.org/jira/browse/FLINK-19717
> > > >> [5]
> > > >>
> > > >>
> > >
> https://issues.apache.org/jira/issues/?jql=project%20%3D%20FLINK%20AND%20priority%20%3D%20Blocker%20AND%20fixVersion%20%3D%201.11.3
> > > >>
> > > >
> > >
>

Re: [DISCUSS] Releasing Apache Flink 1.11.3

Posted by Kostas Kloudas <kk...@apache.org>.
+1 for releasing Flink 1.11.3 as it contains a number of important
fixes and thanks Gordon and Xintong for volunteering.

Cheers,
Kostas

On Mon, Oct 26, 2020 at 4:37 AM Yu Li <ca...@gmail.com> wrote:
>
> +1 for releasing Flink 1.11.3, and thanks Gordon and Xintong for
> volunteering as our release managers.
>
> Best Regards,
> Yu
>
>
> On Mon, 26 Oct 2020 at 09:45, Xintong Song <to...@gmail.com> wrote:
>
> > Thanks Gordan for starting this discussion.
> > My pleasure to help with the release process.
> >
> > Thank you~
> >
> > Xintong Song
> >
> >
> >
> > On Fri, Oct 23, 2020 at 11:29 PM Till Rohrmann <tr...@apache.org>
> > wrote:
> >
> > > Thanks for starting this discussion Gordon. There are over 100 issues
> > > which are fixed for 1.11.3. Hence +1 for a soonish 1.11.3 release. Thanks
> > > for volunteering as our release managers Gordon and Xintong!
> > >
> > > Cheers,
> > > Till
> > >
> > > On Fri, Oct 23, 2020 at 5:02 PM Tzu-Li (Gordon) Tai <tzulitai@apache.org
> > >
> > > wrote:
> > >
> > >> Hi,
> > >>
> > >> Xintong and I would like to start a discussion for releasing Flink
> > 1.11.3
> > >> soon.
> > >>
> > >> It seems like we already have a few pressing issues that needs to be
> > >> included in a new hotfix release:
> > >>
> > >>    - Heap-based timers’ restore behaviour is causing a critical recovery
> > >>    issue for StateFun [1] [2] [3].
> > >>    - There are several robustness issues for the FLIP-27 new source API,
> > >>    such as [4]. We already have some users using the FLIP-27 API with
> > >> 1.11.x,
> > >>    so it would be important to get those fixes in for 1.11.x as well.
> > >>
> > >> Apart from the issues that are already marked as blocker for 1.11.3 in
> > our
> > >> JIRA [5], please let us know in this thread if there is already ongoing
> > >> work for other important fixes that we should try to include.
> > >>
> > >> Xintong and I would like to volunteer for managing this release, and
> > will
> > >> try to communicate the priority of pending blockers over the next few
> > >> days.
> > >> Since the aforementioned issues are quite critical, we’d like to aim
> > >> for a *feature
> > >> freeze by the end of next week (Oct. 30th)* and start the release voting
> > >> process the week after.
> > >> If that is too short of a notice and you might need more time, please
> > let
> > >> us know!
> > >>
> > >> Cheers,
> > >> Gordon
> > >>
> > >> [1] https://issues.apache.org/jira/browse/FLINK-19692
> > >> [2] https://issues.apache.org/jira/browse/FLINK-19741
> > >> [3] https://issues.apache.org/jira/browse/FLINK-19748
> > >> [4] https://issues.apache.org/jira/browse/FLINK-19717
> > >> [5]
> > >>
> > >>
> > https://issues.apache.org/jira/issues/?jql=project%20%3D%20FLINK%20AND%20priority%20%3D%20Blocker%20AND%20fixVersion%20%3D%201.11.3
> > >>
> > >
> >

Re: [DISCUSS] Releasing Apache Flink 1.11.3

Posted by Yu Li <ca...@gmail.com>.
+1 for releasing Flink 1.11.3, and thanks Gordon and Xintong for
volunteering as our release managers.

Best Regards,
Yu


On Mon, 26 Oct 2020 at 09:45, Xintong Song <to...@gmail.com> wrote:

> Thanks Gordan for starting this discussion.
> My pleasure to help with the release process.
>
> Thank you~
>
> Xintong Song
>
>
>
> On Fri, Oct 23, 2020 at 11:29 PM Till Rohrmann <tr...@apache.org>
> wrote:
>
> > Thanks for starting this discussion Gordon. There are over 100 issues
> > which are fixed for 1.11.3. Hence +1 for a soonish 1.11.3 release. Thanks
> > for volunteering as our release managers Gordon and Xintong!
> >
> > Cheers,
> > Till
> >
> > On Fri, Oct 23, 2020 at 5:02 PM Tzu-Li (Gordon) Tai <tzulitai@apache.org
> >
> > wrote:
> >
> >> Hi,
> >>
> >> Xintong and I would like to start a discussion for releasing Flink
> 1.11.3
> >> soon.
> >>
> >> It seems like we already have a few pressing issues that needs to be
> >> included in a new hotfix release:
> >>
> >>    - Heap-based timers’ restore behaviour is causing a critical recovery
> >>    issue for StateFun [1] [2] [3].
> >>    - There are several robustness issues for the FLIP-27 new source API,
> >>    such as [4]. We already have some users using the FLIP-27 API with
> >> 1.11.x,
> >>    so it would be important to get those fixes in for 1.11.x as well.
> >>
> >> Apart from the issues that are already marked as blocker for 1.11.3 in
> our
> >> JIRA [5], please let us know in this thread if there is already ongoing
> >> work for other important fixes that we should try to include.
> >>
> >> Xintong and I would like to volunteer for managing this release, and
> will
> >> try to communicate the priority of pending blockers over the next few
> >> days.
> >> Since the aforementioned issues are quite critical, we’d like to aim
> >> for a *feature
> >> freeze by the end of next week (Oct. 30th)* and start the release voting
> >> process the week after.
> >> If that is too short of a notice and you might need more time, please
> let
> >> us know!
> >>
> >> Cheers,
> >> Gordon
> >>
> >> [1] https://issues.apache.org/jira/browse/FLINK-19692
> >> [2] https://issues.apache.org/jira/browse/FLINK-19741
> >> [3] https://issues.apache.org/jira/browse/FLINK-19748
> >> [4] https://issues.apache.org/jira/browse/FLINK-19717
> >> [5]
> >>
> >>
> https://issues.apache.org/jira/issues/?jql=project%20%3D%20FLINK%20AND%20priority%20%3D%20Blocker%20AND%20fixVersion%20%3D%201.11.3
> >>
> >
>

Re: [DISCUSS] Releasing Apache Flink 1.11.3

Posted by Xintong Song <to...@gmail.com>.
Thanks Gordan for starting this discussion.
My pleasure to help with the release process.

Thank you~

Xintong Song



On Fri, Oct 23, 2020 at 11:29 PM Till Rohrmann <tr...@apache.org> wrote:

> Thanks for starting this discussion Gordon. There are over 100 issues
> which are fixed for 1.11.3. Hence +1 for a soonish 1.11.3 release. Thanks
> for volunteering as our release managers Gordon and Xintong!
>
> Cheers,
> Till
>
> On Fri, Oct 23, 2020 at 5:02 PM Tzu-Li (Gordon) Tai <tz...@apache.org>
> wrote:
>
>> Hi,
>>
>> Xintong and I would like to start a discussion for releasing Flink 1.11.3
>> soon.
>>
>> It seems like we already have a few pressing issues that needs to be
>> included in a new hotfix release:
>>
>>    - Heap-based timers’ restore behaviour is causing a critical recovery
>>    issue for StateFun [1] [2] [3].
>>    - There are several robustness issues for the FLIP-27 new source API,
>>    such as [4]. We already have some users using the FLIP-27 API with
>> 1.11.x,
>>    so it would be important to get those fixes in for 1.11.x as well.
>>
>> Apart from the issues that are already marked as blocker for 1.11.3 in our
>> JIRA [5], please let us know in this thread if there is already ongoing
>> work for other important fixes that we should try to include.
>>
>> Xintong and I would like to volunteer for managing this release, and will
>> try to communicate the priority of pending blockers over the next few
>> days.
>> Since the aforementioned issues are quite critical, we’d like to aim
>> for a *feature
>> freeze by the end of next week (Oct. 30th)* and start the release voting
>> process the week after.
>> If that is too short of a notice and you might need more time, please let
>> us know!
>>
>> Cheers,
>> Gordon
>>
>> [1] https://issues.apache.org/jira/browse/FLINK-19692
>> [2] https://issues.apache.org/jira/browse/FLINK-19741
>> [3] https://issues.apache.org/jira/browse/FLINK-19748
>> [4] https://issues.apache.org/jira/browse/FLINK-19717
>> [5]
>>
>> https://issues.apache.org/jira/issues/?jql=project%20%3D%20FLINK%20AND%20priority%20%3D%20Blocker%20AND%20fixVersion%20%3D%201.11.3
>>
>

Re: [DISCUSS] Releasing Apache Flink 1.11.3

Posted by Till Rohrmann <tr...@apache.org>.
Thanks for starting this discussion Gordon. There are over 100 issues which
are fixed for 1.11.3. Hence +1 for a soonish 1.11.3 release. Thanks for
volunteering as our release managers Gordon and Xintong!

Cheers,
Till

On Fri, Oct 23, 2020 at 5:02 PM Tzu-Li (Gordon) Tai <tz...@apache.org>
wrote:

> Hi,
>
> Xintong and I would like to start a discussion for releasing Flink 1.11.3
> soon.
>
> It seems like we already have a few pressing issues that needs to be
> included in a new hotfix release:
>
>    - Heap-based timers’ restore behaviour is causing a critical recovery
>    issue for StateFun [1] [2] [3].
>    - There are several robustness issues for the FLIP-27 new source API,
>    such as [4]. We already have some users using the FLIP-27 API with
> 1.11.x,
>    so it would be important to get those fixes in for 1.11.x as well.
>
> Apart from the issues that are already marked as blocker for 1.11.3 in our
> JIRA [5], please let us know in this thread if there is already ongoing
> work for other important fixes that we should try to include.
>
> Xintong and I would like to volunteer for managing this release, and will
> try to communicate the priority of pending blockers over the next few days.
> Since the aforementioned issues are quite critical, we’d like to aim
> for a *feature
> freeze by the end of next week (Oct. 30th)* and start the release voting
> process the week after.
> If that is too short of a notice and you might need more time, please let
> us know!
>
> Cheers,
> Gordon
>
> [1] https://issues.apache.org/jira/browse/FLINK-19692
> [2] https://issues.apache.org/jira/browse/FLINK-19741
> [3] https://issues.apache.org/jira/browse/FLINK-19748
> [4] https://issues.apache.org/jira/browse/FLINK-19717
> [5]
>
> https://issues.apache.org/jira/issues/?jql=project%20%3D%20FLINK%20AND%20priority%20%3D%20Blocker%20AND%20fixVersion%20%3D%201.11.3
>