[DISCUSS] Update Flink Roadmap

[Jark Wu <im...@gmail.com>]

Hi all,

Martijn and I would like to initiate a discussion on the Flink roadmap,
which should cover the project's long-term roadmap and the regular update
mechanism.

Xintong has already started a discussion about Flink 2.0 planning. One of
the points raised in that discussion is that we should have a high-level
discussion of the roadmap to present where the project is heading (which
doesn't necessarily need to block the Flink 2.0 planning). Moreover, the
roadmap on the Flink website [1] hasn't been updated for half a year, and
the last update was for the feature radar for the 1.15 release. It has been
2 years since the community discussed Flink's overall roadmap.

I would like to raise two topics for discussion:

1. The new roadmap. This should be an updated version of the current
roadmap[1].
2. A mechanism to regularly discuss and update the roadmap.

To make the first topic discussion more efficient, Martijn and I volunteer
to summarize the ongoing big things of different components and present a
roadmap draft to the community in the next few weeks. This should be a good
starting point for a more detailed discussion.

Regarding the regular update mechanism, there was a proposal in a thread
[2] three years ago to make the release manager responsible for updating
the roadmap. However, it appears that this was not documented as a release
management task [3], and the roadmap update wasn't performed for releases
1.16 and 1.17.

In my opinion, making release managers responsible for keeping the roadmap
up to date is a good idea. Specifically, release managers of release X can
kick off the roadmap update at the beginning of release X, which can be a
joint task with collecting a feature list [4]. Additionally, release
managers of release X-1 can help verify and remove the accomplished items
from the roadmap and update the feature radar.

What do you think? Do you have other ideas?

Best,
Jark & Martijn

[1]: https://flink.apache.org/roadmap.html
[2]: https://lists.apache.org/thread/o0l3cg6yphxwrww0k7215jgtw3yfoybv
[3]:
https://cwiki.apache.org/confluence/display/FLINK/Flink+Release+Management
[4]: https://cwiki.apache.org/confluence/display/FLINK/1.18+Release

Re: [DISCUSS] Update Flink Roadmap

[Jark Wu <im...@gmail.com>]

Thank you all for helping with the roadmap documentation.
I have merged the roadmap pull request.

Cheers,
Jark

On Wed, 23 Aug 2023 at 15:26, Jing Ge <ji...@ververica.com.invalid> wrote:

> Thanks Jark, +1 for the OLAP :-)
>
> Best regards,
> Jing
>
> On Sun, Aug 20, 2023 at 5:04 PM Jark Wu <im...@gmail.com> wrote:
>
> > Hi all,
> >
> > I have addressed another bunch of comments on the Google doc (mainly
> about
> > the OLAP roadmap).
> > And I have opened a pull request for the website:
> > https://github.com/apache/flink-web/pull/672
> >
> > Please help to review it and continue the discussion on the pull request,
> > thanks a lot!
> >
> > Best,
> > Jark
> >
> > On Tue, 15 Aug 2023 at 12:15, Xintong Song <to...@gmail.com>
> wrote:
> >
> > > Thanks for driving this, Jark.
> > >
> > > The current draft looks good to me. I think it is good to open a PR
> with
> > > it. And if there are other comments, we can discuss them during the PR
> > > review.
> > >
> > > I also added a few minor comments in the draft regarding the feature
> > radar.
> > > Those can also be discussed on the PR.
> > >
> > > Best,
> > >
> > > Xintong
> > >
> > >
> > >
> > > On Tue, Aug 15, 2023 at 11:15 AM Shammon FY <zj...@gmail.com> wrote:
> > >
> > > > Hi Jark,
> > > >
> > > > Sounds good and I would love to, thanks! I will involve you and
> > Xingtong
> > > > on the document after updating.
> > > >
> > > > Best,
> > > > Shammon FY
> > > >
> > > >
> > > > On Mon, Aug 14, 2023 at 10:39 PM Jark Wu <im...@gmail.com> wrote:
> > > >
> > > >> Hi Shammon,
> > > >>
> > > >> Sure, could you help to draft a subsection about this in the google
> > doc?
> > > >>
> > > >> Best,
> > > >> Jark
> > > >>
> > > >> 2023年8月14日 20:30，Shammon FY <zj...@gmail.com> 写道：
> > > >>
> > > >> Thanks @Jark for driving the Flink Roadmap.
> > > >>
> > > >> As we discussed olap in the thread [1] and according to the
> > suggestions
> > > >> from @Xingtong Song, could we add a subsection in `Towards Streaming
> > > >> Warehouses` or `Performance` that the short-lived query in Flink
> > Session
> > > >> Cluster is one of the future directions for Flink?
> > > >>
> > > >> Best,
> > > >> Shammon FY
> > > >>
> > > >> On Mon, Aug 14, 2023 at 8:03 PM Jark Wu <im...@gmail.com> wrote:
> > > >>
> > > >>> Thank you everyone for helping polish the roadmap [1].
> > > >>>
> > > >>> I think I have addressed all the comments and we have included all
> > > >>> ongoing
> > > >>> parts of Flink.
> > > >>> Please feel free to take a last look. I'm going to prepare the pull
> > > >>> request
> > > >>> if there are no more concerns.
> > > >>>
> > > >>> Best,
> > > >>> Jark
> > > >>>
> > > >>> [1]:
> > > >>>
> > > >>>
> > >
> >
> https://docs.google.com/document/d/12BDiVKEsY-f7HI3suO_IxwzCmR04QcVqLarXgyJAb7c/edit
> > > >>>
> > > >>> On Sun, 13 Aug 2023 at 13:04, Yuan Mei <yu...@gmail.com>
> > wrote:
> > > >>>
> > > >>> > Sorry for taking so long
> > > >>> >
> > > >>> > I've added a section about Flink Disaggregated State Management
> > > >>> Evolution
> > > >>> > in the attached doc.
> > > >>> >
> > > >>> > I found some of the contents might be overlapped with the
> > > "large-scale
> > > >>> > streaming jobs". So that part might need some changes as well.
> > > >>> >
> > > >>> > Please let me know what you think.
> > > >>> >
> > > >>> > Best
> > > >>> > Yuan
> > > >>> >
> > > >>> > On Mon, Jul 24, 2023 at 12:07 PM Yuan Mei <
> yuanmei.work@gmail.com>
> > > >>> wrote:
> > > >>> >
> > > >>> > > Sorry have missed this email and respond a bit late.
> > > >>> > >
> > > >>> > > I will put a draft for the long-term vision for the state as
> well
> > > as
> > > >>> > > large-scale state support into the roadmap.
> > > >>> > >
> > > >>> > > Best
> > > >>> > > Yuan
> > > >>> > >
> > > >>> > > On Mon, Jul 17, 2023 at 10:34 AM Jark Wu <im...@gmail.com>
> > wrote:
> > > >>> > >
> > > >>> > >> Hi Jiabao,
> > > >>> > >>
> > > >>> > >> Thank you for your suggestions. I have added them to the
> "Going
> > > >>> Beyond a
> > > >>> > >> SQL Stream/Batch Processing Engine" and "Large-Scale State
> Jobs"
> > > >>> > sections.
> > > >>> > >>
> > > >>> > >> Best,
> > > >>> > >> Jark
> > > >>> > >>
> > > >>> > >> On Thu, 13 Jul 2023 at 16:06, Jiabao Sun <
> > jiabao.sun@xtransfer.cn
> > > >>> > >> .invalid>
> > > >>> > >> wrote:
> > > >>> > >>
> > > >>> > >> > Thanks Jark and Martijn for driving this.
> > > >>> > >> >
> > > >>> > >> > There are two suggestions about the Table API:
> > > >>> > >> >
> > > >>> > >> > - Add the JSON type to adapt to the no sql database type.
> > > >>> > >> > - Remove changelog normalize operator for upsert stream.
> > > >>> > >> >
> > > >>> > >> >
> > > >>> > >> > Best,
> > > >>> > >> > Jiabao
> > > >>> > >> >
> > > >>> > >> >
> > > >>> > >> > > 2023年7月13日 下午3:49，Jark Wu <im...@gmail.com> 写道：
> > > >>> > >> > >
> > > >>> > >> > > Hi all,
> > > >>> > >> > >
> > > >>> > >> > > Sorry for taking so long back here.
> > > >>> > >> > >
> > > >>> > >> > > Martijn and I have drafted the first version of the
> updated
> > > >>> roadmap,
> > > >>> > >> > > including the updated feature radar reflecting the current
> > > >>> state of
> > > >>> > >> > > different components.
> > > >>> > >> > >
> > > >>> > >> >
> > > >>> > >>
> > > >>> >
> > > >>>
> > >
> >
> https://docs.google.com/document/d/12BDiVKEsY-f7HI3suO_IxwzCmR04QcVqLarXgyJAb7c/edit
> > > >>> > >> > >
> > > >>> > >> > > Feel free to leave comments in the thread or the document.
> > > >>> > >> > > We may miss mentioning something important, so your help
> in
> > > >>> > enriching
> > > >>> > >> > > the content is greatly appreciated.
> > > >>> > >> > >
> > > >>> > >> > > Best,
> > > >>> > >> > > Jark & Martijn
> > > >>> > >> > >
> > > >>> > >> > >
> > > >>> > >> > > On Fri, 2 Jun 2023 at 00:50, Jing Ge
> > > <jing@ververica.com.invalid
> > > >>> >
> > > >>> > >> wrote:
> > > >>> > >> > >
> > > >>> > >> > >> Hi Jark,
> > > >>> > >> > >>
> > > >>> > >> > >> Fair enough. Let's do it like you suggested. Thanks!
> > > >>> > >> > >>
> > > >>> > >> > >> Best regards,
> > > >>> > >> > >> Jing
> > > >>> > >> > >>
> > > >>> > >> > >> On Thu, Jun 1, 2023 at 6:00 PM Jark Wu <imjark@gmail.com
> >
> > > >>> wrote:
> > > >>> > >> > >>
> > > >>> > >> > >>> Hi Jing,
> > > >>> > >> > >>>
> > > >>> > >> > >>> This thread is for discussing the roadmap for versions
> > 1.18,
> > > >>> 2.0,
> > > >>> > >> and
> > > >>> > >> > >> even
> > > >>> > >> > >>> more.
> > > >>> > >> > >>> One of the outcomes of this discussion will be an
> updated
> > > >>> version
> > > >>> > of
> > > >>> > >> > the
> > > >>> > >> > >>> current roadmap.
> > > >>> > >> > >>> Let's work together on refining the roadmap in this
> > thread.
> > > >>> > >> > >>>
> > > >>> > >> > >>> Best,
> > > >>> > >> > >>> Jark
> > > >>> > >> > >>>
> > > >>> > >> > >>> On Thu, 1 Jun 2023 at 23:25, Jing Ge
> > > >>> <ji...@ververica.com.invalid>
> > > >>> > >> > wrote:
> > > >>> > >> > >>>
> > > >>> > >> > >>>> Hi Jark,
> > > >>> > >> > >>>>
> > > >>> > >> > >>>> Thanks for driving it! For point 2, since we are
> > developing
> > > >>> 1.18
> > > >>> > >> now,
> > > >>> > >> > >>>> does it make sense to update the roadmap this time
> while
> > we
> > > >>> are
> > > >>> > >> > >> releasing
> > > >>> > >> > >>>> 1.18? This discussion thread will be focusing on the
> > Flink
> > > >>> 2.0
> > > >>> > >> > roadmap,
> > > >>> > >> > >>> as
> > > >>> > >> > >>>> you mentioned previously. WDYT?
> > > >>> > >> > >>>>
> > > >>> > >> > >>>> Best regards,
> > > >>> > >> > >>>> Jing
> > > >>> > >> > >>>>
> > > >>> > >> > >>>> On Thu, Jun 1, 2023 at 3:31 PM Jark Wu <
> imjark@gmail.com
> > >
> > > >>> wrote:
> > > >>> > >> > >>>>
> > > >>> > >> > >>>>> Hi all,
> > > >>> > >> > >>>>>
> > > >>> > >> > >>>>> Martijn and I would like to initiate a discussion on
> the
> > > >>> Flink
> > > >>> > >> > >> roadmap,
> > > >>> > >> > >>>>> which should cover the project's long-term roadmap and
> > the
> > > >>> > regular
> > > >>> > >> > >>> update
> > > >>> > >> > >>>>> mechanism.
> > > >>> > >> > >>>>>
> > > >>> > >> > >>>>> Xintong has already started a discussion about Flink
> 2.0
> > > >>> > planning.
> > > >>> > >> > >> One
> > > >>> > >> > >>> of
> > > >>> > >> > >>>>> the points raised in that discussion is that we should
> > > have
> > > >>> a
> > > >>> > >> > >>> high-level
> > > >>> > >> > >>>>> discussion of the roadmap to present where the project
> > is
> > > >>> > heading
> > > >>> > >> > >>> (which
> > > >>> > >> > >>>>> doesn't necessarily need to block the Flink 2.0
> > planning).
> > > >>> > >> Moreover,
> > > >>> > >> > >>> the
> > > >>> > >> > >>>>> roadmap on the Flink website [1] hasn't been updated
> for
> > > >>> half a
> > > >>> > >> year,
> > > >>> > >> > >>> and
> > > >>> > >> > >>>>> the last update was for the feature radar for the 1.15
> > > >>> release.
> > > >>> > It
> > > >>> > >> > >> has
> > > >>> > >> > >>>> been
> > > >>> > >> > >>>>> 2 years since the community discussed Flink's overall
> > > >>> roadmap.
> > > >>> > >> > >>>>>
> > > >>> > >> > >>>>> I would like to raise two topics for discussion:
> > > >>> > >> > >>>>>
> > > >>> > >> > >>>>> 1. The new roadmap. This should be an updated version
> of
> > > the
> > > >>> > >> current
> > > >>> > >> > >>>>> roadmap[1].
> > > >>> > >> > >>>>> 2. A mechanism to regularly discuss and update the
> > > roadmap.
> > > >>> > >> > >>>>>
> > > >>> > >> > >>>>> To make the first topic discussion more efficient,
> > Martijn
> > > >>> and I
> > > >>> > >> > >>>> volunteer
> > > >>> > >> > >>>>> to summarize the ongoing big things of different
> > > components
> > > >>> and
> > > >>> > >> > >>> present a
> > > >>> > >> > >>>>> roadmap draft to the community in the next few weeks.
> > This
> > > >>> > should
> > > >>> > >> be
> > > >>> > >> > >> a
> > > >>> > >> > >>>> good
> > > >>> > >> > >>>>> starting point for a more detailed discussion.
> > > >>> > >> > >>>>>
> > > >>> > >> > >>>>> Regarding the regular update mechanism, there was a
> > > >>> proposal in
> > > >>> > a
> > > >>> > >> > >>> thread
> > > >>> > >> > >>>>> [2] three years ago to make the release manager
> > > responsible
> > > >>> for
> > > >>> > >> > >>> updating
> > > >>> > >> > >>>>> the roadmap. However, it appears that this was not
> > > >>> documented
> > > >>> > as a
> > > >>> > >> > >>>> release
> > > >>> > >> > >>>>> management task [3], and the roadmap update wasn't
> > > >>> performed for
> > > >>> > >> > >>> releases
> > > >>> > >> > >>>>> 1.16 and 1.17.
> > > >>> > >> > >>>>>
> > > >>> > >> > >>>>> In my opinion, making release managers responsible for
> > > >>> keeping
> > > >>> > the
> > > >>> > >> > >>>> roadmap
> > > >>> > >> > >>>>> up to date is a good idea. Specifically, release
> > managers
> > > of
> > > >>> > >> release
> > > >>> > >> > >> X
> > > >>> > >> > >>>> can
> > > >>> > >> > >>>>> kick off the roadmap update at the beginning of
> release
> > X,
> > > >>> which
> > > >>> > >> can
> > > >>> > >> > >>> be a
> > > >>> > >> > >>>>> joint task with collecting a feature list [4].
> > > Additionally,
> > > >>> > >> release
> > > >>> > >> > >>>>> managers of release X-1 can help verify and remove the
> > > >>> > >> accomplished
> > > >>> > >> > >>> items
> > > >>> > >> > >>>>> from the roadmap and update the feature radar.
> > > >>> > >> > >>>>>
> > > >>> > >> > >>>>> What do you think? Do you have other ideas?
> > > >>> > >> > >>>>>
> > > >>> > >> > >>>>> Best,
> > > >>> > >> > >>>>> Jark & Martijn
> > > >>> > >> > >>>>>
> > > >>> > >> > >>>>> [1]: https://flink.apache.org/roadmap.html
> > > >>> > >> > >>>>> [2]:
> > > >>> > >> > >>
> > > >>> https://lists.apache.org/thread/o0l3cg6yphxwrww0k7215jgtw3yfoybv
> > > >>> > >> > >>>>> [3]:
> > > >>> > >> > >>>>>
> > > >>> > >> > >>>>
> > > >>> > >> > >>>
> > > >>> > >> > >>
> > > >>> > >> >
> > > >>> > >>
> > > >>> >
> > > >>>
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/Flink+Release+Management
> > > >>> > >> > >>>>> [4]:
> > > >>> > >>
> https://cwiki.apache.org/confluence/display/FLINK/1.18+Release
> > > >>> > >> > >>>>>
> > > >>> > >> > >>>>
> > > >>> > >> > >>>
> > > >>> > >> > >>
> > > >>> > >> >
> > > >>> > >> >
> > > >>> > >>
> > > >>> > >
> > > >>> >
> > > >>>
> > > >>
> > > >>
> > >
> >
>

Re: [DISCUSS] Update Flink Roadmap

[Jing Ge <ji...@ververica.com.INVALID>]

Thanks Jark, +1 for the OLAP :-)

Best regards,
Jing

On Sun, Aug 20, 2023 at 5:04 PM Jark Wu <im...@gmail.com> wrote:

> Hi all,
>
> I have addressed another bunch of comments on the Google doc (mainly about
> the OLAP roadmap).
> And I have opened a pull request for the website:
> https://github.com/apache/flink-web/pull/672
>
> Please help to review it and continue the discussion on the pull request,
> thanks a lot!
>
> Best,
> Jark
>
> On Tue, 15 Aug 2023 at 12:15, Xintong Song <to...@gmail.com> wrote:
>
> > Thanks for driving this, Jark.
> >
> > The current draft looks good to me. I think it is good to open a PR with
> > it. And if there are other comments, we can discuss them during the PR
> > review.
> >
> > I also added a few minor comments in the draft regarding the feature
> radar.
> > Those can also be discussed on the PR.
> >
> > Best,
> >
> > Xintong
> >
> >
> >
> > On Tue, Aug 15, 2023 at 11:15 AM Shammon FY <zj...@gmail.com> wrote:
> >
> > > Hi Jark,
> > >
> > > Sounds good and I would love to, thanks! I will involve you and
> Xingtong
> > > on the document after updating.
> > >
> > > Best,
> > > Shammon FY
> > >
> > >
> > > On Mon, Aug 14, 2023 at 10:39 PM Jark Wu <im...@gmail.com> wrote:
> > >
> > >> Hi Shammon,
> > >>
> > >> Sure, could you help to draft a subsection about this in the google
> doc?
> > >>
> > >> Best,
> > >> Jark
> > >>
> > >> 2023年8月14日 20:30，Shammon FY <zj...@gmail.com> 写道：
> > >>
> > >> Thanks @Jark for driving the Flink Roadmap.
> > >>
> > >> As we discussed olap in the thread [1] and according to the
> suggestions
> > >> from @Xingtong Song, could we add a subsection in `Towards Streaming
> > >> Warehouses` or `Performance` that the short-lived query in Flink
> Session
> > >> Cluster is one of the future directions for Flink?
> > >>
> > >> Best,
> > >> Shammon FY
> > >>
> > >> On Mon, Aug 14, 2023 at 8:03 PM Jark Wu <im...@gmail.com> wrote:
> > >>
> > >>> Thank you everyone for helping polish the roadmap [1].
> > >>>
> > >>> I think I have addressed all the comments and we have included all
> > >>> ongoing
> > >>> parts of Flink.
> > >>> Please feel free to take a last look. I'm going to prepare the pull
> > >>> request
> > >>> if there are no more concerns.
> > >>>
> > >>> Best,
> > >>> Jark
> > >>>
> > >>> [1]:
> > >>>
> > >>>
> >
> https://docs.google.com/document/d/12BDiVKEsY-f7HI3suO_IxwzCmR04QcVqLarXgyJAb7c/edit
> > >>>
> > >>> On Sun, 13 Aug 2023 at 13:04, Yuan Mei <yu...@gmail.com>
> wrote:
> > >>>
> > >>> > Sorry for taking so long
> > >>> >
> > >>> > I've added a section about Flink Disaggregated State Management
> > >>> Evolution
> > >>> > in the attached doc.
> > >>> >
> > >>> > I found some of the contents might be overlapped with the
> > "large-scale
> > >>> > streaming jobs". So that part might need some changes as well.
> > >>> >
> > >>> > Please let me know what you think.
> > >>> >
> > >>> > Best
> > >>> > Yuan
> > >>> >
> > >>> > On Mon, Jul 24, 2023 at 12:07 PM Yuan Mei <yu...@gmail.com>
> > >>> wrote:
> > >>> >
> > >>> > > Sorry have missed this email and respond a bit late.
> > >>> > >
> > >>> > > I will put a draft for the long-term vision for the state as well
> > as
> > >>> > > large-scale state support into the roadmap.
> > >>> > >
> > >>> > > Best
> > >>> > > Yuan
> > >>> > >
> > >>> > > On Mon, Jul 17, 2023 at 10:34 AM Jark Wu <im...@gmail.com>
> wrote:
> > >>> > >
> > >>> > >> Hi Jiabao,
> > >>> > >>
> > >>> > >> Thank you for your suggestions. I have added them to the "Going
> > >>> Beyond a
> > >>> > >> SQL Stream/Batch Processing Engine" and "Large-Scale State Jobs"
> > >>> > sections.
> > >>> > >>
> > >>> > >> Best,
> > >>> > >> Jark
> > >>> > >>
> > >>> > >> On Thu, 13 Jul 2023 at 16:06, Jiabao Sun <
> jiabao.sun@xtransfer.cn
> > >>> > >> .invalid>
> > >>> > >> wrote:
> > >>> > >>
> > >>> > >> > Thanks Jark and Martijn for driving this.
> > >>> > >> >
> > >>> > >> > There are two suggestions about the Table API:
> > >>> > >> >
> > >>> > >> > - Add the JSON type to adapt to the no sql database type.
> > >>> > >> > - Remove changelog normalize operator for upsert stream.
> > >>> > >> >
> > >>> > >> >
> > >>> > >> > Best,
> > >>> > >> > Jiabao
> > >>> > >> >
> > >>> > >> >
> > >>> > >> > > 2023年7月13日 下午3:49，Jark Wu <im...@gmail.com> 写道：
> > >>> > >> > >
> > >>> > >> > > Hi all,
> > >>> > >> > >
> > >>> > >> > > Sorry for taking so long back here.
> > >>> > >> > >
> > >>> > >> > > Martijn and I have drafted the first version of the updated
> > >>> roadmap,
> > >>> > >> > > including the updated feature radar reflecting the current
> > >>> state of
> > >>> > >> > > different components.
> > >>> > >> > >
> > >>> > >> >
> > >>> > >>
> > >>> >
> > >>>
> >
> https://docs.google.com/document/d/12BDiVKEsY-f7HI3suO_IxwzCmR04QcVqLarXgyJAb7c/edit
> > >>> > >> > >
> > >>> > >> > > Feel free to leave comments in the thread or the document.
> > >>> > >> > > We may miss mentioning something important, so your help in
> > >>> > enriching
> > >>> > >> > > the content is greatly appreciated.
> > >>> > >> > >
> > >>> > >> > > Best,
> > >>> > >> > > Jark & Martijn
> > >>> > >> > >
> > >>> > >> > >
> > >>> > >> > > On Fri, 2 Jun 2023 at 00:50, Jing Ge
> > <jing@ververica.com.invalid
> > >>> >
> > >>> > >> wrote:
> > >>> > >> > >
> > >>> > >> > >> Hi Jark,
> > >>> > >> > >>
> > >>> > >> > >> Fair enough. Let's do it like you suggested. Thanks!
> > >>> > >> > >>
> > >>> > >> > >> Best regards,
> > >>> > >> > >> Jing
> > >>> > >> > >>
> > >>> > >> > >> On Thu, Jun 1, 2023 at 6:00 PM Jark Wu <im...@gmail.com>
> > >>> wrote:
> > >>> > >> > >>
> > >>> > >> > >>> Hi Jing,
> > >>> > >> > >>>
> > >>> > >> > >>> This thread is for discussing the roadmap for versions
> 1.18,
> > >>> 2.0,
> > >>> > >> and
> > >>> > >> > >> even
> > >>> > >> > >>> more.
> > >>> > >> > >>> One of the outcomes of this discussion will be an updated
> > >>> version
> > >>> > of
> > >>> > >> > the
> > >>> > >> > >>> current roadmap.
> > >>> > >> > >>> Let's work together on refining the roadmap in this
> thread.
> > >>> > >> > >>>
> > >>> > >> > >>> Best,
> > >>> > >> > >>> Jark
> > >>> > >> > >>>
> > >>> > >> > >>> On Thu, 1 Jun 2023 at 23:25, Jing Ge
> > >>> <ji...@ververica.com.invalid>
> > >>> > >> > wrote:
> > >>> > >> > >>>
> > >>> > >> > >>>> Hi Jark,
> > >>> > >> > >>>>
> > >>> > >> > >>>> Thanks for driving it! For point 2, since we are
> developing
> > >>> 1.18
> > >>> > >> now,
> > >>> > >> > >>>> does it make sense to update the roadmap this time while
> we
> > >>> are
> > >>> > >> > >> releasing
> > >>> > >> > >>>> 1.18? This discussion thread will be focusing on the
> Flink
> > >>> 2.0
> > >>> > >> > roadmap,
> > >>> > >> > >>> as
> > >>> > >> > >>>> you mentioned previously. WDYT?
> > >>> > >> > >>>>
> > >>> > >> > >>>> Best regards,
> > >>> > >> > >>>> Jing
> > >>> > >> > >>>>
> > >>> > >> > >>>> On Thu, Jun 1, 2023 at 3:31 PM Jark Wu <imjark@gmail.com
> >
> > >>> wrote:
> > >>> > >> > >>>>
> > >>> > >> > >>>>> Hi all,
> > >>> > >> > >>>>>
> > >>> > >> > >>>>> Martijn and I would like to initiate a discussion on the
> > >>> Flink
> > >>> > >> > >> roadmap,
> > >>> > >> > >>>>> which should cover the project's long-term roadmap and
> the
> > >>> > regular
> > >>> > >> > >>> update
> > >>> > >> > >>>>> mechanism.
> > >>> > >> > >>>>>
> > >>> > >> > >>>>> Xintong has already started a discussion about Flink 2.0
> > >>> > planning.
> > >>> > >> > >> One
> > >>> > >> > >>> of
> > >>> > >> > >>>>> the points raised in that discussion is that we should
> > have
> > >>> a
> > >>> > >> > >>> high-level
> > >>> > >> > >>>>> discussion of the roadmap to present where the project
> is
> > >>> > heading
> > >>> > >> > >>> (which
> > >>> > >> > >>>>> doesn't necessarily need to block the Flink 2.0
> planning).
> > >>> > >> Moreover,
> > >>> > >> > >>> the
> > >>> > >> > >>>>> roadmap on the Flink website [1] hasn't been updated for
> > >>> half a
> > >>> > >> year,
> > >>> > >> > >>> and
> > >>> > >> > >>>>> the last update was for the feature radar for the 1.15
> > >>> release.
> > >>> > It
> > >>> > >> > >> has
> > >>> > >> > >>>> been
> > >>> > >> > >>>>> 2 years since the community discussed Flink's overall
> > >>> roadmap.
> > >>> > >> > >>>>>
> > >>> > >> > >>>>> I would like to raise two topics for discussion:
> > >>> > >> > >>>>>
> > >>> > >> > >>>>> 1. The new roadmap. This should be an updated version of
> > the
> > >>> > >> current
> > >>> > >> > >>>>> roadmap[1].
> > >>> > >> > >>>>> 2. A mechanism to regularly discuss and update the
> > roadmap.
> > >>> > >> > >>>>>
> > >>> > >> > >>>>> To make the first topic discussion more efficient,
> Martijn
> > >>> and I
> > >>> > >> > >>>> volunteer
> > >>> > >> > >>>>> to summarize the ongoing big things of different
> > components
> > >>> and
> > >>> > >> > >>> present a
> > >>> > >> > >>>>> roadmap draft to the community in the next few weeks.
> This
> > >>> > should
> > >>> > >> be
> > >>> > >> > >> a
> > >>> > >> > >>>> good
> > >>> > >> > >>>>> starting point for a more detailed discussion.
> > >>> > >> > >>>>>
> > >>> > >> > >>>>> Regarding the regular update mechanism, there was a
> > >>> proposal in
> > >>> > a
> > >>> > >> > >>> thread
> > >>> > >> > >>>>> [2] three years ago to make the release manager
> > responsible
> > >>> for
> > >>> > >> > >>> updating
> > >>> > >> > >>>>> the roadmap. However, it appears that this was not
> > >>> documented
> > >>> > as a
> > >>> > >> > >>>> release
> > >>> > >> > >>>>> management task [3], and the roadmap update wasn't
> > >>> performed for
> > >>> > >> > >>> releases
> > >>> > >> > >>>>> 1.16 and 1.17.
> > >>> > >> > >>>>>
> > >>> > >> > >>>>> In my opinion, making release managers responsible for
> > >>> keeping
> > >>> > the
> > >>> > >> > >>>> roadmap
> > >>> > >> > >>>>> up to date is a good idea. Specifically, release
> managers
> > of
> > >>> > >> release
> > >>> > >> > >> X
> > >>> > >> > >>>> can
> > >>> > >> > >>>>> kick off the roadmap update at the beginning of release
> X,
> > >>> which
> > >>> > >> can
> > >>> > >> > >>> be a
> > >>> > >> > >>>>> joint task with collecting a feature list [4].
> > Additionally,
> > >>> > >> release
> > >>> > >> > >>>>> managers of release X-1 can help verify and remove the
> > >>> > >> accomplished
> > >>> > >> > >>> items
> > >>> > >> > >>>>> from the roadmap and update the feature radar.
> > >>> > >> > >>>>>
> > >>> > >> > >>>>> What do you think? Do you have other ideas?
> > >>> > >> > >>>>>
> > >>> > >> > >>>>> Best,
> > >>> > >> > >>>>> Jark & Martijn
> > >>> > >> > >>>>>
> > >>> > >> > >>>>> [1]: https://flink.apache.org/roadmap.html
> > >>> > >> > >>>>> [2]:
> > >>> > >> > >>
> > >>> https://lists.apache.org/thread/o0l3cg6yphxwrww0k7215jgtw3yfoybv
> > >>> > >> > >>>>> [3]:
> > >>> > >> > >>>>>
> > >>> > >> > >>>>
> > >>> > >> > >>>
> > >>> > >> > >>
> > >>> > >> >
> > >>> > >>
> > >>> >
> > >>>
> >
> https://cwiki.apache.org/confluence/display/FLINK/Flink+Release+Management
> > >>> > >> > >>>>> [4]:
> > >>> > >> https://cwiki.apache.org/confluence/display/FLINK/1.18+Release
> > >>> > >> > >>>>>
> > >>> > >> > >>>>
> > >>> > >> > >>>
> > >>> > >> > >>
> > >>> > >> >
> > >>> > >> >
> > >>> > >>
> > >>> > >
> > >>> >
> > >>>
> > >>
> > >>
> >
>

Re: [DISCUSS] Update Flink Roadmap

[ConradJam <ja...@gmail.com>]

+1 Thanks for all suggest. It looks great that in feature

Jark Wu <im...@gmail.com> 于2023年8月20日周日 23:03写道：

> Hi all,
>
> I have addressed another bunch of comments on the Google doc (mainly about
> the OLAP roadmap).
> And I have opened a pull request for the website:
> https://github.com/apache/flink-web/pull/672
>
> Please help to review it and continue the discussion on the pull request,
> thanks a lot!
>
> Best,
> Jark
>
> On Tue, 15 Aug 2023 at 12:15, Xintong Song <to...@gmail.com> wrote:
>
> > Thanks for driving this, Jark.
> >
> > The current draft looks good to me. I think it is good to open a PR with
> > it. And if there are other comments, we can discuss them during the PR
> > review.
> >
> > I also added a few minor comments in the draft regarding the feature
> radar.
> > Those can also be discussed on the PR.
> >
> > Best,
> >
> > Xintong
> >
> >
> >
> > On Tue, Aug 15, 2023 at 11:15 AM Shammon FY <zj...@gmail.com> wrote:
> >
> > > Hi Jark,
> > >
> > > Sounds good and I would love to, thanks! I will involve you and
> Xingtong
> > > on the document after updating.
> > >
> > > Best,
> > > Shammon FY
> > >
> > >
> > > On Mon, Aug 14, 2023 at 10:39 PM Jark Wu <im...@gmail.com> wrote:
> > >
> > >> Hi Shammon,
> > >>
> > >> Sure, could you help to draft a subsection about this in the google
> doc?
> > >>
> > >> Best,
> > >> Jark
> > >>
> > >> 2023年8月14日 20:30，Shammon FY <zj...@gmail.com> 写道：
> > >>
> > >> Thanks @Jark for driving the Flink Roadmap.
> > >>
> > >> As we discussed olap in the thread [1] and according to the
> suggestions
> > >> from @Xingtong Song, could we add a subsection in `Towards Streaming
> > >> Warehouses` or `Performance` that the short-lived query in Flink
> Session
> > >> Cluster is one of the future directions for Flink?
> > >>
> > >> Best,
> > >> Shammon FY
> > >>
> > >> On Mon, Aug 14, 2023 at 8:03 PM Jark Wu <im...@gmail.com> wrote:
> > >>
> > >>> Thank you everyone for helping polish the roadmap [1].
> > >>>
> > >>> I think I have addressed all the comments and we have included all
> > >>> ongoing
> > >>> parts of Flink.
> > >>> Please feel free to take a last look. I'm going to prepare the pull
> > >>> request
> > >>> if there are no more concerns.
> > >>>
> > >>> Best,
> > >>> Jark
> > >>>
> > >>> [1]:
> > >>>
> > >>>
> >
> https://docs.google.com/document/d/12BDiVKEsY-f7HI3suO_IxwzCmR04QcVqLarXgyJAb7c/edit
> > >>>
> > >>> On Sun, 13 Aug 2023 at 13:04, Yuan Mei <yu...@gmail.com>
> wrote:
> > >>>
> > >>> > Sorry for taking so long
> > >>> >
> > >>> > I've added a section about Flink Disaggregated State Management
> > >>> Evolution
> > >>> > in the attached doc.
> > >>> >
> > >>> > I found some of the contents might be overlapped with the
> > "large-scale
> > >>> > streaming jobs". So that part might need some changes as well.
> > >>> >
> > >>> > Please let me know what you think.
> > >>> >
> > >>> > Best
> > >>> > Yuan
> > >>> >
> > >>> > On Mon, Jul 24, 2023 at 12:07 PM Yuan Mei <yu...@gmail.com>
> > >>> wrote:
> > >>> >
> > >>> > > Sorry have missed this email and respond a bit late.
> > >>> > >
> > >>> > > I will put a draft for the long-term vision for the state as well
> > as
> > >>> > > large-scale state support into the roadmap.
> > >>> > >
> > >>> > > Best
> > >>> > > Yuan
> > >>> > >
> > >>> > > On Mon, Jul 17, 2023 at 10:34 AM Jark Wu <im...@gmail.com>
> wrote:
> > >>> > >
> > >>> > >> Hi Jiabao,
> > >>> > >>
> > >>> > >> Thank you for your suggestions. I have added them to the "Going
> > >>> Beyond a
> > >>> > >> SQL Stream/Batch Processing Engine" and "Large-Scale State Jobs"
> > >>> > sections.
> > >>> > >>
> > >>> > >> Best,
> > >>> > >> Jark
> > >>> > >>
> > >>> > >> On Thu, 13 Jul 2023 at 16:06, Jiabao Sun <
> jiabao.sun@xtransfer.cn
> > >>> > >> .invalid>
> > >>> > >> wrote:
> > >>> > >>
> > >>> > >> > Thanks Jark and Martijn for driving this.
> > >>> > >> >
> > >>> > >> > There are two suggestions about the Table API:
> > >>> > >> >
> > >>> > >> > - Add the JSON type to adapt to the no sql database type.
> > >>> > >> > - Remove changelog normalize operator for upsert stream.
> > >>> > >> >
> > >>> > >> >
> > >>> > >> > Best,
> > >>> > >> > Jiabao
> > >>> > >> >
> > >>> > >> >
> > >>> > >> > > 2023年7月13日 下午3:49，Jark Wu <im...@gmail.com> 写道：
> > >>> > >> > >
> > >>> > >> > > Hi all,
> > >>> > >> > >
> > >>> > >> > > Sorry for taking so long back here.
> > >>> > >> > >
> > >>> > >> > > Martijn and I have drafted the first version of the updated
> > >>> roadmap,
> > >>> > >> > > including the updated feature radar reflecting the current
> > >>> state of
> > >>> > >> > > different components.
> > >>> > >> > >
> > >>> > >> >
> > >>> > >>
> > >>> >
> > >>>
> >
> https://docs.google.com/document/d/12BDiVKEsY-f7HI3suO_IxwzCmR04QcVqLarXgyJAb7c/edit
> > >>> > >> > >
> > >>> > >> > > Feel free to leave comments in the thread or the document.
> > >>> > >> > > We may miss mentioning something important, so your help in
> > >>> > enriching
> > >>> > >> > > the content is greatly appreciated.
> > >>> > >> > >
> > >>> > >> > > Best,
> > >>> > >> > > Jark & Martijn
> > >>> > >> > >
> > >>> > >> > >
> > >>> > >> > > On Fri, 2 Jun 2023 at 00:50, Jing Ge
> > <jing@ververica.com.invalid
> > >>> >
> > >>> > >> wrote:
> > >>> > >> > >
> > >>> > >> > >> Hi Jark,
> > >>> > >> > >>
> > >>> > >> > >> Fair enough. Let's do it like you suggested. Thanks!
> > >>> > >> > >>
> > >>> > >> > >> Best regards,
> > >>> > >> > >> Jing
> > >>> > >> > >>
> > >>> > >> > >> On Thu, Jun 1, 2023 at 6:00 PM Jark Wu <im...@gmail.com>
> > >>> wrote:
> > >>> > >> > >>
> > >>> > >> > >>> Hi Jing,
> > >>> > >> > >>>
> > >>> > >> > >>> This thread is for discussing the roadmap for versions
> 1.18,
> > >>> 2.0,
> > >>> > >> and
> > >>> > >> > >> even
> > >>> > >> > >>> more.
> > >>> > >> > >>> One of the outcomes of this discussion will be an updated
> > >>> version
> > >>> > of
> > >>> > >> > the
> > >>> > >> > >>> current roadmap.
> > >>> > >> > >>> Let's work together on refining the roadmap in this
> thread.
> > >>> > >> > >>>
> > >>> > >> > >>> Best,
> > >>> > >> > >>> Jark
> > >>> > >> > >>>
> > >>> > >> > >>> On Thu, 1 Jun 2023 at 23:25, Jing Ge
> > >>> <ji...@ververica.com.invalid>
> > >>> > >> > wrote:
> > >>> > >> > >>>
> > >>> > >> > >>>> Hi Jark,
> > >>> > >> > >>>>
> > >>> > >> > >>>> Thanks for driving it! For point 2, since we are
> developing
> > >>> 1.18
> > >>> > >> now,
> > >>> > >> > >>>> does it make sense to update the roadmap this time while
> we
> > >>> are
> > >>> > >> > >> releasing
> > >>> > >> > >>>> 1.18? This discussion thread will be focusing on the
> Flink
> > >>> 2.0
> > >>> > >> > roadmap,
> > >>> > >> > >>> as
> > >>> > >> > >>>> you mentioned previously. WDYT?
> > >>> > >> > >>>>
> > >>> > >> > >>>> Best regards,
> > >>> > >> > >>>> Jing
> > >>> > >> > >>>>
> > >>> > >> > >>>> On Thu, Jun 1, 2023 at 3:31 PM Jark Wu <imjark@gmail.com
> >
> > >>> wrote:
> > >>> > >> > >>>>
> > >>> > >> > >>>>> Hi all,
> > >>> > >> > >>>>>
> > >>> > >> > >>>>> Martijn and I would like to initiate a discussion on the
> > >>> Flink
> > >>> > >> > >> roadmap,
> > >>> > >> > >>>>> which should cover the project's long-term roadmap and
> the
> > >>> > regular
> > >>> > >> > >>> update
> > >>> > >> > >>>>> mechanism.
> > >>> > >> > >>>>>
> > >>> > >> > >>>>> Xintong has already started a discussion about Flink 2.0
> > >>> > planning.
> > >>> > >> > >> One
> > >>> > >> > >>> of
> > >>> > >> > >>>>> the points raised in that discussion is that we should
> > have
> > >>> a
> > >>> > >> > >>> high-level
> > >>> > >> > >>>>> discussion of the roadmap to present where the project
> is
> > >>> > heading
> > >>> > >> > >>> (which
> > >>> > >> > >>>>> doesn't necessarily need to block the Flink 2.0
> planning).
> > >>> > >> Moreover,
> > >>> > >> > >>> the
> > >>> > >> > >>>>> roadmap on the Flink website [1] hasn't been updated for
> > >>> half a
> > >>> > >> year,
> > >>> > >> > >>> and
> > >>> > >> > >>>>> the last update was for the feature radar for the 1.15
> > >>> release.
> > >>> > It
> > >>> > >> > >> has
> > >>> > >> > >>>> been
> > >>> > >> > >>>>> 2 years since the community discussed Flink's overall
> > >>> roadmap.
> > >>> > >> > >>>>>
> > >>> > >> > >>>>> I would like to raise two topics for discussion:
> > >>> > >> > >>>>>
> > >>> > >> > >>>>> 1. The new roadmap. This should be an updated version of
> > the
> > >>> > >> current
> > >>> > >> > >>>>> roadmap[1].
> > >>> > >> > >>>>> 2. A mechanism to regularly discuss and update the
> > roadmap.
> > >>> > >> > >>>>>
> > >>> > >> > >>>>> To make the first topic discussion more efficient,
> Martijn
> > >>> and I
> > >>> > >> > >>>> volunteer
> > >>> > >> > >>>>> to summarize the ongoing big things of different
> > components
> > >>> and
> > >>> > >> > >>> present a
> > >>> > >> > >>>>> roadmap draft to the community in the next few weeks.
> This
> > >>> > should
> > >>> > >> be
> > >>> > >> > >> a
> > >>> > >> > >>>> good
> > >>> > >> > >>>>> starting point for a more detailed discussion.
> > >>> > >> > >>>>>
> > >>> > >> > >>>>> Regarding the regular update mechanism, there was a
> > >>> proposal in
> > >>> > a
> > >>> > >> > >>> thread
> > >>> > >> > >>>>> [2] three years ago to make the release manager
> > responsible
> > >>> for
> > >>> > >> > >>> updating
> > >>> > >> > >>>>> the roadmap. However, it appears that this was not
> > >>> documented
> > >>> > as a
> > >>> > >> > >>>> release
> > >>> > >> > >>>>> management task [3], and the roadmap update wasn't
> > >>> performed for
> > >>> > >> > >>> releases
> > >>> > >> > >>>>> 1.16 and 1.17.
> > >>> > >> > >>>>>
> > >>> > >> > >>>>> In my opinion, making release managers responsible for
> > >>> keeping
> > >>> > the
> > >>> > >> > >>>> roadmap
> > >>> > >> > >>>>> up to date is a good idea. Specifically, release
> managers
> > of
> > >>> > >> release
> > >>> > >> > >> X
> > >>> > >> > >>>> can
> > >>> > >> > >>>>> kick off the roadmap update at the beginning of release
> X,
> > >>> which
> > >>> > >> can
> > >>> > >> > >>> be a
> > >>> > >> > >>>>> joint task with collecting a feature list [4].
> > Additionally,
> > >>> > >> release
> > >>> > >> > >>>>> managers of release X-1 can help verify and remove the
> > >>> > >> accomplished
> > >>> > >> > >>> items
> > >>> > >> > >>>>> from the roadmap and update the feature radar.
> > >>> > >> > >>>>>
> > >>> > >> > >>>>> What do you think? Do you have other ideas?
> > >>> > >> > >>>>>
> > >>> > >> > >>>>> Best,
> > >>> > >> > >>>>> Jark & Martijn
> > >>> > >> > >>>>>
> > >>> > >> > >>>>> [1]: https://flink.apache.org/roadmap.html
> > >>> > >> > >>>>> [2]:
> > >>> > >> > >>
> > >>> https://lists.apache.org/thread/o0l3cg6yphxwrww0k7215jgtw3yfoybv
> > >>> > >> > >>>>> [3]:
> > >>> > >> > >>>>>
> > >>> > >> > >>>>
> > >>> > >> > >>>
> > >>> > >> > >>
> > >>> > >> >
> > >>> > >>
> > >>> >
> > >>>
> >
> https://cwiki.apache.org/confluence/display/FLINK/Flink+Release+Management
> > >>> > >> > >>>>> [4]:
> > >>> > >> https://cwiki.apache.org/confluence/display/FLINK/1.18+Release
> > >>> > >> > >>>>>
> > >>> > >> > >>>>
> > >>> > >> > >>>
> > >>> > >> > >>
> > >>> > >> >
> > >>> > >> >
> > >>> > >>
> > >>> > >
> > >>> >
> > >>>
> > >>
> > >>
> >
>


-- 
Best

ConradJam

Re: [DISCUSS] Update Flink Roadmap

[Jark Wu <im...@gmail.com>]

Hi all,

I have addressed another bunch of comments on the Google doc (mainly about
the OLAP roadmap).
And I have opened a pull request for the website:
https://github.com/apache/flink-web/pull/672

Please help to review it and continue the discussion on the pull request,
thanks a lot!

Best,
Jark

On Tue, 15 Aug 2023 at 12:15, Xintong Song <to...@gmail.com> wrote:

> Thanks for driving this, Jark.
>
> The current draft looks good to me. I think it is good to open a PR with
> it. And if there are other comments, we can discuss them during the PR
> review.
>
> I also added a few minor comments in the draft regarding the feature radar.
> Those can also be discussed on the PR.
>
> Best,
>
> Xintong
>
>
>
> On Tue, Aug 15, 2023 at 11:15 AM Shammon FY <zj...@gmail.com> wrote:
>
> > Hi Jark,
> >
> > Sounds good and I would love to, thanks! I will involve you and Xingtong
> > on the document after updating.
> >
> > Best,
> > Shammon FY
> >
> >
> > On Mon, Aug 14, 2023 at 10:39 PM Jark Wu <im...@gmail.com> wrote:
> >
> >> Hi Shammon,
> >>
> >> Sure, could you help to draft a subsection about this in the google doc?
> >>
> >> Best,
> >> Jark
> >>
> >> 2023年8月14日 20:30，Shammon FY <zj...@gmail.com> 写道：
> >>
> >> Thanks @Jark for driving the Flink Roadmap.
> >>
> >> As we discussed olap in the thread [1] and according to the suggestions
> >> from @Xingtong Song, could we add a subsection in `Towards Streaming
> >> Warehouses` or `Performance` that the short-lived query in Flink Session
> >> Cluster is one of the future directions for Flink?
> >>
> >> Best,
> >> Shammon FY
> >>
> >> On Mon, Aug 14, 2023 at 8:03 PM Jark Wu <im...@gmail.com> wrote:
> >>
> >>> Thank you everyone for helping polish the roadmap [1].
> >>>
> >>> I think I have addressed all the comments and we have included all
> >>> ongoing
> >>> parts of Flink.
> >>> Please feel free to take a last look. I'm going to prepare the pull
> >>> request
> >>> if there are no more concerns.
> >>>
> >>> Best,
> >>> Jark
> >>>
> >>> [1]:
> >>>
> >>>
> https://docs.google.com/document/d/12BDiVKEsY-f7HI3suO_IxwzCmR04QcVqLarXgyJAb7c/edit
> >>>
> >>> On Sun, 13 Aug 2023 at 13:04, Yuan Mei <yu...@gmail.com> wrote:
> >>>
> >>> > Sorry for taking so long
> >>> >
> >>> > I've added a section about Flink Disaggregated State Management
> >>> Evolution
> >>> > in the attached doc.
> >>> >
> >>> > I found some of the contents might be overlapped with the
> "large-scale
> >>> > streaming jobs". So that part might need some changes as well.
> >>> >
> >>> > Please let me know what you think.
> >>> >
> >>> > Best
> >>> > Yuan
> >>> >
> >>> > On Mon, Jul 24, 2023 at 12:07 PM Yuan Mei <yu...@gmail.com>
> >>> wrote:
> >>> >
> >>> > > Sorry have missed this email and respond a bit late.
> >>> > >
> >>> > > I will put a draft for the long-term vision for the state as well
> as
> >>> > > large-scale state support into the roadmap.
> >>> > >
> >>> > > Best
> >>> > > Yuan
> >>> > >
> >>> > > On Mon, Jul 17, 2023 at 10:34 AM Jark Wu <im...@gmail.com> wrote:
> >>> > >
> >>> > >> Hi Jiabao,
> >>> > >>
> >>> > >> Thank you for your suggestions. I have added them to the "Going
> >>> Beyond a
> >>> > >> SQL Stream/Batch Processing Engine" and "Large-Scale State Jobs"
> >>> > sections.
> >>> > >>
> >>> > >> Best,
> >>> > >> Jark
> >>> > >>
> >>> > >> On Thu, 13 Jul 2023 at 16:06, Jiabao Sun <jiabao.sun@xtransfer.cn
> >>> > >> .invalid>
> >>> > >> wrote:
> >>> > >>
> >>> > >> > Thanks Jark and Martijn for driving this.
> >>> > >> >
> >>> > >> > There are two suggestions about the Table API:
> >>> > >> >
> >>> > >> > - Add the JSON type to adapt to the no sql database type.
> >>> > >> > - Remove changelog normalize operator for upsert stream.
> >>> > >> >
> >>> > >> >
> >>> > >> > Best,
> >>> > >> > Jiabao
> >>> > >> >
> >>> > >> >
> >>> > >> > > 2023年7月13日 下午3:49，Jark Wu <im...@gmail.com> 写道：
> >>> > >> > >
> >>> > >> > > Hi all,
> >>> > >> > >
> >>> > >> > > Sorry for taking so long back here.
> >>> > >> > >
> >>> > >> > > Martijn and I have drafted the first version of the updated
> >>> roadmap,
> >>> > >> > > including the updated feature radar reflecting the current
> >>> state of
> >>> > >> > > different components.
> >>> > >> > >
> >>> > >> >
> >>> > >>
> >>> >
> >>>
> https://docs.google.com/document/d/12BDiVKEsY-f7HI3suO_IxwzCmR04QcVqLarXgyJAb7c/edit
> >>> > >> > >
> >>> > >> > > Feel free to leave comments in the thread or the document.
> >>> > >> > > We may miss mentioning something important, so your help in
> >>> > enriching
> >>> > >> > > the content is greatly appreciated.
> >>> > >> > >
> >>> > >> > > Best,
> >>> > >> > > Jark & Martijn
> >>> > >> > >
> >>> > >> > >
> >>> > >> > > On Fri, 2 Jun 2023 at 00:50, Jing Ge
> <jing@ververica.com.invalid
> >>> >
> >>> > >> wrote:
> >>> > >> > >
> >>> > >> > >> Hi Jark,
> >>> > >> > >>
> >>> > >> > >> Fair enough. Let's do it like you suggested. Thanks!
> >>> > >> > >>
> >>> > >> > >> Best regards,
> >>> > >> > >> Jing
> >>> > >> > >>
> >>> > >> > >> On Thu, Jun 1, 2023 at 6:00 PM Jark Wu <im...@gmail.com>
> >>> wrote:
> >>> > >> > >>
> >>> > >> > >>> Hi Jing,
> >>> > >> > >>>
> >>> > >> > >>> This thread is for discussing the roadmap for versions 1.18,
> >>> 2.0,
> >>> > >> and
> >>> > >> > >> even
> >>> > >> > >>> more.
> >>> > >> > >>> One of the outcomes of this discussion will be an updated
> >>> version
> >>> > of
> >>> > >> > the
> >>> > >> > >>> current roadmap.
> >>> > >> > >>> Let's work together on refining the roadmap in this thread.
> >>> > >> > >>>
> >>> > >> > >>> Best,
> >>> > >> > >>> Jark
> >>> > >> > >>>
> >>> > >> > >>> On Thu, 1 Jun 2023 at 23:25, Jing Ge
> >>> <ji...@ververica.com.invalid>
> >>> > >> > wrote:
> >>> > >> > >>>
> >>> > >> > >>>> Hi Jark,
> >>> > >> > >>>>
> >>> > >> > >>>> Thanks for driving it! For point 2, since we are developing
> >>> 1.18
> >>> > >> now,
> >>> > >> > >>>> does it make sense to update the roadmap this time while we
> >>> are
> >>> > >> > >> releasing
> >>> > >> > >>>> 1.18? This discussion thread will be focusing on the Flink
> >>> 2.0
> >>> > >> > roadmap,
> >>> > >> > >>> as
> >>> > >> > >>>> you mentioned previously. WDYT?
> >>> > >> > >>>>
> >>> > >> > >>>> Best regards,
> >>> > >> > >>>> Jing
> >>> > >> > >>>>
> >>> > >> > >>>> On Thu, Jun 1, 2023 at 3:31 PM Jark Wu <im...@gmail.com>
> >>> wrote:
> >>> > >> > >>>>
> >>> > >> > >>>>> Hi all,
> >>> > >> > >>>>>
> >>> > >> > >>>>> Martijn and I would like to initiate a discussion on the
> >>> Flink
> >>> > >> > >> roadmap,
> >>> > >> > >>>>> which should cover the project's long-term roadmap and the
> >>> > regular
> >>> > >> > >>> update
> >>> > >> > >>>>> mechanism.
> >>> > >> > >>>>>
> >>> > >> > >>>>> Xintong has already started a discussion about Flink 2.0
> >>> > planning.
> >>> > >> > >> One
> >>> > >> > >>> of
> >>> > >> > >>>>> the points raised in that discussion is that we should
> have
> >>> a
> >>> > >> > >>> high-level
> >>> > >> > >>>>> discussion of the roadmap to present where the project is
> >>> > heading
> >>> > >> > >>> (which
> >>> > >> > >>>>> doesn't necessarily need to block the Flink 2.0 planning).
> >>> > >> Moreover,
> >>> > >> > >>> the
> >>> > >> > >>>>> roadmap on the Flink website [1] hasn't been updated for
> >>> half a
> >>> > >> year,
> >>> > >> > >>> and
> >>> > >> > >>>>> the last update was for the feature radar for the 1.15
> >>> release.
> >>> > It
> >>> > >> > >> has
> >>> > >> > >>>> been
> >>> > >> > >>>>> 2 years since the community discussed Flink's overall
> >>> roadmap.
> >>> > >> > >>>>>
> >>> > >> > >>>>> I would like to raise two topics for discussion:
> >>> > >> > >>>>>
> >>> > >> > >>>>> 1. The new roadmap. This should be an updated version of
> the
> >>> > >> current
> >>> > >> > >>>>> roadmap[1].
> >>> > >> > >>>>> 2. A mechanism to regularly discuss and update the
> roadmap.
> >>> > >> > >>>>>
> >>> > >> > >>>>> To make the first topic discussion more efficient, Martijn
> >>> and I
> >>> > >> > >>>> volunteer
> >>> > >> > >>>>> to summarize the ongoing big things of different
> components
> >>> and
> >>> > >> > >>> present a
> >>> > >> > >>>>> roadmap draft to the community in the next few weeks. This
> >>> > should
> >>> > >> be
> >>> > >> > >> a
> >>> > >> > >>>> good
> >>> > >> > >>>>> starting point for a more detailed discussion.
> >>> > >> > >>>>>
> >>> > >> > >>>>> Regarding the regular update mechanism, there was a
> >>> proposal in
> >>> > a
> >>> > >> > >>> thread
> >>> > >> > >>>>> [2] three years ago to make the release manager
> responsible
> >>> for
> >>> > >> > >>> updating
> >>> > >> > >>>>> the roadmap. However, it appears that this was not
> >>> documented
> >>> > as a
> >>> > >> > >>>> release
> >>> > >> > >>>>> management task [3], and the roadmap update wasn't
> >>> performed for
> >>> > >> > >>> releases
> >>> > >> > >>>>> 1.16 and 1.17.
> >>> > >> > >>>>>
> >>> > >> > >>>>> In my opinion, making release managers responsible for
> >>> keeping
> >>> > the
> >>> > >> > >>>> roadmap
> >>> > >> > >>>>> up to date is a good idea. Specifically, release managers
> of
> >>> > >> release
> >>> > >> > >> X
> >>> > >> > >>>> can
> >>> > >> > >>>>> kick off the roadmap update at the beginning of release X,
> >>> which
> >>> > >> can
> >>> > >> > >>> be a
> >>> > >> > >>>>> joint task with collecting a feature list [4].
> Additionally,
> >>> > >> release
> >>> > >> > >>>>> managers of release X-1 can help verify and remove the
> >>> > >> accomplished
> >>> > >> > >>> items
> >>> > >> > >>>>> from the roadmap and update the feature radar.
> >>> > >> > >>>>>
> >>> > >> > >>>>> What do you think? Do you have other ideas?
> >>> > >> > >>>>>
> >>> > >> > >>>>> Best,
> >>> > >> > >>>>> Jark & Martijn
> >>> > >> > >>>>>
> >>> > >> > >>>>> [1]: https://flink.apache.org/roadmap.html
> >>> > >> > >>>>> [2]:
> >>> > >> > >>
> >>> https://lists.apache.org/thread/o0l3cg6yphxwrww0k7215jgtw3yfoybv
> >>> > >> > >>>>> [3]:
> >>> > >> > >>>>>
> >>> > >> > >>>>
> >>> > >> > >>>
> >>> > >> > >>
> >>> > >> >
> >>> > >>
> >>> >
> >>>
> https://cwiki.apache.org/confluence/display/FLINK/Flink+Release+Management
> >>> > >> > >>>>> [4]:
> >>> > >> https://cwiki.apache.org/confluence/display/FLINK/1.18+Release
> >>> > >> > >>>>>
> >>> > >> > >>>>
> >>> > >> > >>>
> >>> > >> > >>
> >>> > >> >
> >>> > >> >
> >>> > >>
> >>> > >
> >>> >
> >>>
> >>
> >>
>

Re: [DISCUSS] Update Flink Roadmap

[Xintong Song <to...@gmail.com>]

Thanks for driving this, Jark.

The current draft looks good to me. I think it is good to open a PR with
it. And if there are other comments, we can discuss them during the PR
review.

I also added a few minor comments in the draft regarding the feature radar.
Those can also be discussed on the PR.

Best,

Xintong



On Tue, Aug 15, 2023 at 11:15 AM Shammon FY <zj...@gmail.com> wrote:

> Hi Jark,
>
> Sounds good and I would love to, thanks! I will involve you and Xingtong
> on the document after updating.
>
> Best,
> Shammon FY
>
>
> On Mon, Aug 14, 2023 at 10:39 PM Jark Wu <im...@gmail.com> wrote:
>
>> Hi Shammon,
>>
>> Sure, could you help to draft a subsection about this in the google doc?
>>
>> Best,
>> Jark
>>
>> 2023年8月14日 20:30，Shammon FY <zj...@gmail.com> 写道：
>>
>> Thanks @Jark for driving the Flink Roadmap.
>>
>> As we discussed olap in the thread [1] and according to the suggestions
>> from @Xingtong Song, could we add a subsection in `Towards Streaming
>> Warehouses` or `Performance` that the short-lived query in Flink Session
>> Cluster is one of the future directions for Flink?
>>
>> Best,
>> Shammon FY
>>
>> On Mon, Aug 14, 2023 at 8:03 PM Jark Wu <im...@gmail.com> wrote:
>>
>>> Thank you everyone for helping polish the roadmap [1].
>>>
>>> I think I have addressed all the comments and we have included all
>>> ongoing
>>> parts of Flink.
>>> Please feel free to take a last look. I'm going to prepare the pull
>>> request
>>> if there are no more concerns.
>>>
>>> Best,
>>> Jark
>>>
>>> [1]:
>>>
>>> https://docs.google.com/document/d/12BDiVKEsY-f7HI3suO_IxwzCmR04QcVqLarXgyJAb7c/edit
>>>
>>> On Sun, 13 Aug 2023 at 13:04, Yuan Mei <yu...@gmail.com> wrote:
>>>
>>> > Sorry for taking so long
>>> >
>>> > I've added a section about Flink Disaggregated State Management
>>> Evolution
>>> > in the attached doc.
>>> >
>>> > I found some of the contents might be overlapped with the "large-scale
>>> > streaming jobs". So that part might need some changes as well.
>>> >
>>> > Please let me know what you think.
>>> >
>>> > Best
>>> > Yuan
>>> >
>>> > On Mon, Jul 24, 2023 at 12:07 PM Yuan Mei <yu...@gmail.com>
>>> wrote:
>>> >
>>> > > Sorry have missed this email and respond a bit late.
>>> > >
>>> > > I will put a draft for the long-term vision for the state as well as
>>> > > large-scale state support into the roadmap.
>>> > >
>>> > > Best
>>> > > Yuan
>>> > >
>>> > > On Mon, Jul 17, 2023 at 10:34 AM Jark Wu <im...@gmail.com> wrote:
>>> > >
>>> > >> Hi Jiabao,
>>> > >>
>>> > >> Thank you for your suggestions. I have added them to the "Going
>>> Beyond a
>>> > >> SQL Stream/Batch Processing Engine" and "Large-Scale State Jobs"
>>> > sections.
>>> > >>
>>> > >> Best,
>>> > >> Jark
>>> > >>
>>> > >> On Thu, 13 Jul 2023 at 16:06, Jiabao Sun <jiabao.sun@xtransfer.cn
>>> > >> .invalid>
>>> > >> wrote:
>>> > >>
>>> > >> > Thanks Jark and Martijn for driving this.
>>> > >> >
>>> > >> > There are two suggestions about the Table API:
>>> > >> >
>>> > >> > - Add the JSON type to adapt to the no sql database type.
>>> > >> > - Remove changelog normalize operator for upsert stream.
>>> > >> >
>>> > >> >
>>> > >> > Best,
>>> > >> > Jiabao
>>> > >> >
>>> > >> >
>>> > >> > > 2023年7月13日 下午3:49，Jark Wu <im...@gmail.com> 写道：
>>> > >> > >
>>> > >> > > Hi all,
>>> > >> > >
>>> > >> > > Sorry for taking so long back here.
>>> > >> > >
>>> > >> > > Martijn and I have drafted the first version of the updated
>>> roadmap,
>>> > >> > > including the updated feature radar reflecting the current
>>> state of
>>> > >> > > different components.
>>> > >> > >
>>> > >> >
>>> > >>
>>> >
>>> https://docs.google.com/document/d/12BDiVKEsY-f7HI3suO_IxwzCmR04QcVqLarXgyJAb7c/edit
>>> > >> > >
>>> > >> > > Feel free to leave comments in the thread or the document.
>>> > >> > > We may miss mentioning something important, so your help in
>>> > enriching
>>> > >> > > the content is greatly appreciated.
>>> > >> > >
>>> > >> > > Best,
>>> > >> > > Jark & Martijn
>>> > >> > >
>>> > >> > >
>>> > >> > > On Fri, 2 Jun 2023 at 00:50, Jing Ge <jing@ververica.com.invalid
>>> >
>>> > >> wrote:
>>> > >> > >
>>> > >> > >> Hi Jark,
>>> > >> > >>
>>> > >> > >> Fair enough. Let's do it like you suggested. Thanks!
>>> > >> > >>
>>> > >> > >> Best regards,
>>> > >> > >> Jing
>>> > >> > >>
>>> > >> > >> On Thu, Jun 1, 2023 at 6:00 PM Jark Wu <im...@gmail.com>
>>> wrote:
>>> > >> > >>
>>> > >> > >>> Hi Jing,
>>> > >> > >>>
>>> > >> > >>> This thread is for discussing the roadmap for versions 1.18,
>>> 2.0,
>>> > >> and
>>> > >> > >> even
>>> > >> > >>> more.
>>> > >> > >>> One of the outcomes of this discussion will be an updated
>>> version
>>> > of
>>> > >> > the
>>> > >> > >>> current roadmap.
>>> > >> > >>> Let's work together on refining the roadmap in this thread.
>>> > >> > >>>
>>> > >> > >>> Best,
>>> > >> > >>> Jark
>>> > >> > >>>
>>> > >> > >>> On Thu, 1 Jun 2023 at 23:25, Jing Ge
>>> <ji...@ververica.com.invalid>
>>> > >> > wrote:
>>> > >> > >>>
>>> > >> > >>>> Hi Jark,
>>> > >> > >>>>
>>> > >> > >>>> Thanks for driving it! For point 2, since we are developing
>>> 1.18
>>> > >> now,
>>> > >> > >>>> does it make sense to update the roadmap this time while we
>>> are
>>> > >> > >> releasing
>>> > >> > >>>> 1.18? This discussion thread will be focusing on the Flink
>>> 2.0
>>> > >> > roadmap,
>>> > >> > >>> as
>>> > >> > >>>> you mentioned previously. WDYT?
>>> > >> > >>>>
>>> > >> > >>>> Best regards,
>>> > >> > >>>> Jing
>>> > >> > >>>>
>>> > >> > >>>> On Thu, Jun 1, 2023 at 3:31 PM Jark Wu <im...@gmail.com>
>>> wrote:
>>> > >> > >>>>
>>> > >> > >>>>> Hi all,
>>> > >> > >>>>>
>>> > >> > >>>>> Martijn and I would like to initiate a discussion on the
>>> Flink
>>> > >> > >> roadmap,
>>> > >> > >>>>> which should cover the project's long-term roadmap and the
>>> > regular
>>> > >> > >>> update
>>> > >> > >>>>> mechanism.
>>> > >> > >>>>>
>>> > >> > >>>>> Xintong has already started a discussion about Flink 2.0
>>> > planning.
>>> > >> > >> One
>>> > >> > >>> of
>>> > >> > >>>>> the points raised in that discussion is that we should have
>>> a
>>> > >> > >>> high-level
>>> > >> > >>>>> discussion of the roadmap to present where the project is
>>> > heading
>>> > >> > >>> (which
>>> > >> > >>>>> doesn't necessarily need to block the Flink 2.0 planning).
>>> > >> Moreover,
>>> > >> > >>> the
>>> > >> > >>>>> roadmap on the Flink website [1] hasn't been updated for
>>> half a
>>> > >> year,
>>> > >> > >>> and
>>> > >> > >>>>> the last update was for the feature radar for the 1.15
>>> release.
>>> > It
>>> > >> > >> has
>>> > >> > >>>> been
>>> > >> > >>>>> 2 years since the community discussed Flink's overall
>>> roadmap.
>>> > >> > >>>>>
>>> > >> > >>>>> I would like to raise two topics for discussion:
>>> > >> > >>>>>
>>> > >> > >>>>> 1. The new roadmap. This should be an updated version of the
>>> > >> current
>>> > >> > >>>>> roadmap[1].
>>> > >> > >>>>> 2. A mechanism to regularly discuss and update the roadmap.
>>> > >> > >>>>>
>>> > >> > >>>>> To make the first topic discussion more efficient, Martijn
>>> and I
>>> > >> > >>>> volunteer
>>> > >> > >>>>> to summarize the ongoing big things of different components
>>> and
>>> > >> > >>> present a
>>> > >> > >>>>> roadmap draft to the community in the next few weeks. This
>>> > should
>>> > >> be
>>> > >> > >> a
>>> > >> > >>>> good
>>> > >> > >>>>> starting point for a more detailed discussion.
>>> > >> > >>>>>
>>> > >> > >>>>> Regarding the regular update mechanism, there was a
>>> proposal in
>>> > a
>>> > >> > >>> thread
>>> > >> > >>>>> [2] three years ago to make the release manager responsible
>>> for
>>> > >> > >>> updating
>>> > >> > >>>>> the roadmap. However, it appears that this was not
>>> documented
>>> > as a
>>> > >> > >>>> release
>>> > >> > >>>>> management task [3], and the roadmap update wasn't
>>> performed for
>>> > >> > >>> releases
>>> > >> > >>>>> 1.16 and 1.17.
>>> > >> > >>>>>
>>> > >> > >>>>> In my opinion, making release managers responsible for
>>> keeping
>>> > the
>>> > >> > >>>> roadmap
>>> > >> > >>>>> up to date is a good idea. Specifically, release managers of
>>> > >> release
>>> > >> > >> X
>>> > >> > >>>> can
>>> > >> > >>>>> kick off the roadmap update at the beginning of release X,
>>> which
>>> > >> can
>>> > >> > >>> be a
>>> > >> > >>>>> joint task with collecting a feature list [4]. Additionally,
>>> > >> release
>>> > >> > >>>>> managers of release X-1 can help verify and remove the
>>> > >> accomplished
>>> > >> > >>> items
>>> > >> > >>>>> from the roadmap and update the feature radar.
>>> > >> > >>>>>
>>> > >> > >>>>> What do you think? Do you have other ideas?
>>> > >> > >>>>>
>>> > >> > >>>>> Best,
>>> > >> > >>>>> Jark & Martijn
>>> > >> > >>>>>
>>> > >> > >>>>> [1]: https://flink.apache.org/roadmap.html
>>> > >> > >>>>> [2]:
>>> > >> > >>
>>> https://lists.apache.org/thread/o0l3cg6yphxwrww0k7215jgtw3yfoybv
>>> > >> > >>>>> [3]:
>>> > >> > >>>>>
>>> > >> > >>>>
>>> > >> > >>>
>>> > >> > >>
>>> > >> >
>>> > >>
>>> >
>>> https://cwiki.apache.org/confluence/display/FLINK/Flink+Release+Management
>>> > >> > >>>>> [4]:
>>> > >> https://cwiki.apache.org/confluence/display/FLINK/1.18+Release
>>> > >> > >>>>>
>>> > >> > >>>>
>>> > >> > >>>
>>> > >> > >>
>>> > >> >
>>> > >> >
>>> > >>
>>> > >
>>> >
>>>
>>
>>

Re: [DISCUSS] Update Flink Roadmap

[Shammon FY <zj...@gmail.com>]

Hi Jark,

Sounds good and I would love to, thanks! I will involve you and Xingtong on
the document after updating.

Best,
Shammon FY


On Mon, Aug 14, 2023 at 10:39 PM Jark Wu <im...@gmail.com> wrote:

> Hi Shammon,
>
> Sure, could you help to draft a subsection about this in the google doc?
>
> Best,
> Jark
>
> 2023年8月14日 20:30，Shammon FY <zj...@gmail.com> 写道：
>
> Thanks @Jark for driving the Flink Roadmap.
>
> As we discussed olap in the thread [1] and according to the suggestions
> from @Xingtong Song, could we add a subsection in `Towards Streaming
> Warehouses` or `Performance` that the short-lived query in Flink Session
> Cluster is one of the future directions for Flink?
>
> Best,
> Shammon FY
>
> On Mon, Aug 14, 2023 at 8:03 PM Jark Wu <im...@gmail.com> wrote:
>
>> Thank you everyone for helping polish the roadmap [1].
>>
>> I think I have addressed all the comments and we have included all ongoing
>> parts of Flink.
>> Please feel free to take a last look. I'm going to prepare the pull
>> request
>> if there are no more concerns.
>>
>> Best,
>> Jark
>>
>> [1]:
>>
>> https://docs.google.com/document/d/12BDiVKEsY-f7HI3suO_IxwzCmR04QcVqLarXgyJAb7c/edit
>>
>> On Sun, 13 Aug 2023 at 13:04, Yuan Mei <yu...@gmail.com> wrote:
>>
>> > Sorry for taking so long
>> >
>> > I've added a section about Flink Disaggregated State Management
>> Evolution
>> > in the attached doc.
>> >
>> > I found some of the contents might be overlapped with the "large-scale
>> > streaming jobs". So that part might need some changes as well.
>> >
>> > Please let me know what you think.
>> >
>> > Best
>> > Yuan
>> >
>> > On Mon, Jul 24, 2023 at 12:07 PM Yuan Mei <yu...@gmail.com>
>> wrote:
>> >
>> > > Sorry have missed this email and respond a bit late.
>> > >
>> > > I will put a draft for the long-term vision for the state as well as
>> > > large-scale state support into the roadmap.
>> > >
>> > > Best
>> > > Yuan
>> > >
>> > > On Mon, Jul 17, 2023 at 10:34 AM Jark Wu <im...@gmail.com> wrote:
>> > >
>> > >> Hi Jiabao,
>> > >>
>> > >> Thank you for your suggestions. I have added them to the "Going
>> Beyond a
>> > >> SQL Stream/Batch Processing Engine" and "Large-Scale State Jobs"
>> > sections.
>> > >>
>> > >> Best,
>> > >> Jark
>> > >>
>> > >> On Thu, 13 Jul 2023 at 16:06, Jiabao Sun <jiabao.sun@xtransfer.cn
>> > >> .invalid>
>> > >> wrote:
>> > >>
>> > >> > Thanks Jark and Martijn for driving this.
>> > >> >
>> > >> > There are two suggestions about the Table API:
>> > >> >
>> > >> > - Add the JSON type to adapt to the no sql database type.
>> > >> > - Remove changelog normalize operator for upsert stream.
>> > >> >
>> > >> >
>> > >> > Best,
>> > >> > Jiabao
>> > >> >
>> > >> >
>> > >> > > 2023年7月13日 下午3:49，Jark Wu <im...@gmail.com> 写道：
>> > >> > >
>> > >> > > Hi all,
>> > >> > >
>> > >> > > Sorry for taking so long back here.
>> > >> > >
>> > >> > > Martijn and I have drafted the first version of the updated
>> roadmap,
>> > >> > > including the updated feature radar reflecting the current state
>> of
>> > >> > > different components.
>> > >> > >
>> > >> >
>> > >>
>> >
>> https://docs.google.com/document/d/12BDiVKEsY-f7HI3suO_IxwzCmR04QcVqLarXgyJAb7c/edit
>> > >> > >
>> > >> > > Feel free to leave comments in the thread or the document.
>> > >> > > We may miss mentioning something important, so your help in
>> > enriching
>> > >> > > the content is greatly appreciated.
>> > >> > >
>> > >> > > Best,
>> > >> > > Jark & Martijn
>> > >> > >
>> > >> > >
>> > >> > > On Fri, 2 Jun 2023 at 00:50, Jing Ge <jing@ververica.com.invalid
>> >
>> > >> wrote:
>> > >> > >
>> > >> > >> Hi Jark,
>> > >> > >>
>> > >> > >> Fair enough. Let's do it like you suggested. Thanks!
>> > >> > >>
>> > >> > >> Best regards,
>> > >> > >> Jing
>> > >> > >>
>> > >> > >> On Thu, Jun 1, 2023 at 6:00 PM Jark Wu <im...@gmail.com>
>> wrote:
>> > >> > >>
>> > >> > >>> Hi Jing,
>> > >> > >>>
>> > >> > >>> This thread is for discussing the roadmap for versions 1.18,
>> 2.0,
>> > >> and
>> > >> > >> even
>> > >> > >>> more.
>> > >> > >>> One of the outcomes of this discussion will be an updated
>> version
>> > of
>> > >> > the
>> > >> > >>> current roadmap.
>> > >> > >>> Let's work together on refining the roadmap in this thread.
>> > >> > >>>
>> > >> > >>> Best,
>> > >> > >>> Jark
>> > >> > >>>
>> > >> > >>> On Thu, 1 Jun 2023 at 23:25, Jing Ge
>> <ji...@ververica.com.invalid>
>> > >> > wrote:
>> > >> > >>>
>> > >> > >>>> Hi Jark,
>> > >> > >>>>
>> > >> > >>>> Thanks for driving it! For point 2, since we are developing
>> 1.18
>> > >> now,
>> > >> > >>>> does it make sense to update the roadmap this time while we
>> are
>> > >> > >> releasing
>> > >> > >>>> 1.18? This discussion thread will be focusing on the Flink 2.0
>> > >> > roadmap,
>> > >> > >>> as
>> > >> > >>>> you mentioned previously. WDYT?
>> > >> > >>>>
>> > >> > >>>> Best regards,
>> > >> > >>>> Jing
>> > >> > >>>>
>> > >> > >>>> On Thu, Jun 1, 2023 at 3:31 PM Jark Wu <im...@gmail.com>
>> wrote:
>> > >> > >>>>
>> > >> > >>>>> Hi all,
>> > >> > >>>>>
>> > >> > >>>>> Martijn and I would like to initiate a discussion on the
>> Flink
>> > >> > >> roadmap,
>> > >> > >>>>> which should cover the project's long-term roadmap and the
>> > regular
>> > >> > >>> update
>> > >> > >>>>> mechanism.
>> > >> > >>>>>
>> > >> > >>>>> Xintong has already started a discussion about Flink 2.0
>> > planning.
>> > >> > >> One
>> > >> > >>> of
>> > >> > >>>>> the points raised in that discussion is that we should have a
>> > >> > >>> high-level
>> > >> > >>>>> discussion of the roadmap to present where the project is
>> > heading
>> > >> > >>> (which
>> > >> > >>>>> doesn't necessarily need to block the Flink 2.0 planning).
>> > >> Moreover,
>> > >> > >>> the
>> > >> > >>>>> roadmap on the Flink website [1] hasn't been updated for
>> half a
>> > >> year,
>> > >> > >>> and
>> > >> > >>>>> the last update was for the feature radar for the 1.15
>> release.
>> > It
>> > >> > >> has
>> > >> > >>>> been
>> > >> > >>>>> 2 years since the community discussed Flink's overall
>> roadmap.
>> > >> > >>>>>
>> > >> > >>>>> I would like to raise two topics for discussion:
>> > >> > >>>>>
>> > >> > >>>>> 1. The new roadmap. This should be an updated version of the
>> > >> current
>> > >> > >>>>> roadmap[1].
>> > >> > >>>>> 2. A mechanism to regularly discuss and update the roadmap.
>> > >> > >>>>>
>> > >> > >>>>> To make the first topic discussion more efficient, Martijn
>> and I
>> > >> > >>>> volunteer
>> > >> > >>>>> to summarize the ongoing big things of different components
>> and
>> > >> > >>> present a
>> > >> > >>>>> roadmap draft to the community in the next few weeks. This
>> > should
>> > >> be
>> > >> > >> a
>> > >> > >>>> good
>> > >> > >>>>> starting point for a more detailed discussion.
>> > >> > >>>>>
>> > >> > >>>>> Regarding the regular update mechanism, there was a proposal
>> in
>> > a
>> > >> > >>> thread
>> > >> > >>>>> [2] three years ago to make the release manager responsible
>> for
>> > >> > >>> updating
>> > >> > >>>>> the roadmap. However, it appears that this was not documented
>> > as a
>> > >> > >>>> release
>> > >> > >>>>> management task [3], and the roadmap update wasn't performed
>> for
>> > >> > >>> releases
>> > >> > >>>>> 1.16 and 1.17.
>> > >> > >>>>>
>> > >> > >>>>> In my opinion, making release managers responsible for
>> keeping
>> > the
>> > >> > >>>> roadmap
>> > >> > >>>>> up to date is a good idea. Specifically, release managers of
>> > >> release
>> > >> > >> X
>> > >> > >>>> can
>> > >> > >>>>> kick off the roadmap update at the beginning of release X,
>> which
>> > >> can
>> > >> > >>> be a
>> > >> > >>>>> joint task with collecting a feature list [4]. Additionally,
>> > >> release
>> > >> > >>>>> managers of release X-1 can help verify and remove the
>> > >> accomplished
>> > >> > >>> items
>> > >> > >>>>> from the roadmap and update the feature radar.
>> > >> > >>>>>
>> > >> > >>>>> What do you think? Do you have other ideas?
>> > >> > >>>>>
>> > >> > >>>>> Best,
>> > >> > >>>>> Jark & Martijn
>> > >> > >>>>>
>> > >> > >>>>> [1]: https://flink.apache.org/roadmap.html
>> > >> > >>>>> [2]:
>> > >> > >>
>> https://lists.apache.org/thread/o0l3cg6yphxwrww0k7215jgtw3yfoybv
>> > >> > >>>>> [3]:
>> > >> > >>>>>
>> > >> > >>>>
>> > >> > >>>
>> > >> > >>
>> > >> >
>> > >>
>> >
>> https://cwiki.apache.org/confluence/display/FLINK/Flink+Release+Management
>> > >> > >>>>> [4]:
>> > >> https://cwiki.apache.org/confluence/display/FLINK/1.18+Release
>> > >> > >>>>>
>> > >> > >>>>
>> > >> > >>>
>> > >> > >>
>> > >> >
>> > >> >
>> > >>
>> > >
>> >
>>
>
>

Re: [DISCUSS] Update Flink Roadmap

[Jark Wu <im...@gmail.com>]

Hi Shammon,

Sure, could you help to draft a subsection about this in the google doc?

Best,
Jark

> 2023年8月14日 20:30，Shammon FY <zj...@gmail.com> 写道：
> 
> Thanks @Jark for driving the Flink Roadmap. 
> 
> As we discussed olap in the thread [1] and according to the suggestions from @Xingtong Song, could we add a subsection in `Towards Streaming Warehouses` or `Performance` that the short-lived query in Flink Session Cluster is one of the future directions for Flink?
> 
> Best,
> Shammon FY
> 
> On Mon, Aug 14, 2023 at 8:03 PM Jark Wu <imjark@gmail.com <ma...@gmail.com>> wrote:
>> Thank you everyone for helping polish the roadmap [1].
>> 
>> I think I have addressed all the comments and we have included all ongoing
>> parts of Flink.
>> Please feel free to take a last look. I'm going to prepare the pull request
>> if there are no more concerns.
>> 
>> Best,
>> Jark
>> 
>> [1]:
>> https://docs.google.com/document/d/12BDiVKEsY-f7HI3suO_IxwzCmR04QcVqLarXgyJAb7c/edit
>> 
>> On Sun, 13 Aug 2023 at 13:04, Yuan Mei <yuanmei.work@gmail.com <ma...@gmail.com>> wrote:
>> 
>> > Sorry for taking so long
>> >
>> > I've added a section about Flink Disaggregated State Management Evolution
>> > in the attached doc.
>> >
>> > I found some of the contents might be overlapped with the "large-scale
>> > streaming jobs". So that part might need some changes as well.
>> >
>> > Please let me know what you think.
>> >
>> > Best
>> > Yuan
>> >
>> > On Mon, Jul 24, 2023 at 12:07 PM Yuan Mei <yuanmei.work@gmail.com <ma...@gmail.com>> wrote:
>> >
>> > > Sorry have missed this email and respond a bit late.
>> > >
>> > > I will put a draft for the long-term vision for the state as well as
>> > > large-scale state support into the roadmap.
>> > >
>> > > Best
>> > > Yuan
>> > >
>> > > On Mon, Jul 17, 2023 at 10:34 AM Jark Wu <imjark@gmail.com <ma...@gmail.com>> wrote:
>> > >
>> > >> Hi Jiabao,
>> > >>
>> > >> Thank you for your suggestions. I have added them to the "Going Beyond a
>> > >> SQL Stream/Batch Processing Engine" and "Large-Scale State Jobs"
>> > sections.
>> > >>
>> > >> Best,
>> > >> Jark
>> > >>
>> > >> On Thu, 13 Jul 2023 at 16:06, Jiabao Sun <jiabao.sun@xtransfer.cn <ma...@xtransfer.cn>
>> > >> .invalid>
>> > >> wrote:
>> > >>
>> > >> > Thanks Jark and Martijn for driving this.
>> > >> >
>> > >> > There are two suggestions about the Table API:
>> > >> >
>> > >> > - Add the JSON type to adapt to the no sql database type.
>> > >> > - Remove changelog normalize operator for upsert stream.
>> > >> >
>> > >> >
>> > >> > Best,
>> > >> > Jiabao
>> > >> >
>> > >> >
>> > >> > > 2023年7月13日 下午3:49，Jark Wu <imjark@gmail.com <ma...@gmail.com>> 写道：
>> > >> > >
>> > >> > > Hi all,
>> > >> > >
>> > >> > > Sorry for taking so long back here.
>> > >> > >
>> > >> > > Martijn and I have drafted the first version of the updated roadmap,
>> > >> > > including the updated feature radar reflecting the current state of
>> > >> > > different components.
>> > >> > >
>> > >> >
>> > >>
>> > https://docs.google.com/document/d/12BDiVKEsY-f7HI3suO_IxwzCmR04QcVqLarXgyJAb7c/edit
>> > >> > >
>> > >> > > Feel free to leave comments in the thread or the document.
>> > >> > > We may miss mentioning something important, so your help in
>> > enriching
>> > >> > > the content is greatly appreciated.
>> > >> > >
>> > >> > > Best,
>> > >> > > Jark & Martijn
>> > >> > >
>> > >> > >
>> > >> > > On Fri, 2 Jun 2023 at 00:50, Jing Ge <ji...@ververica.com.invalid>
>> > >> wrote:
>> > >> > >
>> > >> > >> Hi Jark,
>> > >> > >>
>> > >> > >> Fair enough. Let's do it like you suggested. Thanks!
>> > >> > >>
>> > >> > >> Best regards,
>> > >> > >> Jing
>> > >> > >>
>> > >> > >> On Thu, Jun 1, 2023 at 6:00 PM Jark Wu <imjark@gmail.com <ma...@gmail.com>> wrote:
>> > >> > >>
>> > >> > >>> Hi Jing,
>> > >> > >>>
>> > >> > >>> This thread is for discussing the roadmap for versions 1.18, 2.0,
>> > >> and
>> > >> > >> even
>> > >> > >>> more.
>> > >> > >>> One of the outcomes of this discussion will be an updated version
>> > of
>> > >> > the
>> > >> > >>> current roadmap.
>> > >> > >>> Let's work together on refining the roadmap in this thread.
>> > >> > >>>
>> > >> > >>> Best,
>> > >> > >>> Jark
>> > >> > >>>
>> > >> > >>> On Thu, 1 Jun 2023 at 23:25, Jing Ge <ji...@ververica.com.invalid>
>> > >> > wrote:
>> > >> > >>>
>> > >> > >>>> Hi Jark,
>> > >> > >>>>
>> > >> > >>>> Thanks for driving it! For point 2, since we are developing 1.18
>> > >> now,
>> > >> > >>>> does it make sense to update the roadmap this time while we are
>> > >> > >> releasing
>> > >> > >>>> 1.18? This discussion thread will be focusing on the Flink 2.0
>> > >> > roadmap,
>> > >> > >>> as
>> > >> > >>>> you mentioned previously. WDYT?
>> > >> > >>>>
>> > >> > >>>> Best regards,
>> > >> > >>>> Jing
>> > >> > >>>>
>> > >> > >>>> On Thu, Jun 1, 2023 at 3:31 PM Jark Wu <imjark@gmail.com <ma...@gmail.com>> wrote:
>> > >> > >>>>
>> > >> > >>>>> Hi all,
>> > >> > >>>>>
>> > >> > >>>>> Martijn and I would like to initiate a discussion on the Flink
>> > >> > >> roadmap,
>> > >> > >>>>> which should cover the project's long-term roadmap and the
>> > regular
>> > >> > >>> update
>> > >> > >>>>> mechanism.
>> > >> > >>>>>
>> > >> > >>>>> Xintong has already started a discussion about Flink 2.0
>> > planning.
>> > >> > >> One
>> > >> > >>> of
>> > >> > >>>>> the points raised in that discussion is that we should have a
>> > >> > >>> high-level
>> > >> > >>>>> discussion of the roadmap to present where the project is
>> > heading
>> > >> > >>> (which
>> > >> > >>>>> doesn't necessarily need to block the Flink 2.0 planning).
>> > >> Moreover,
>> > >> > >>> the
>> > >> > >>>>> roadmap on the Flink website [1] hasn't been updated for half a
>> > >> year,
>> > >> > >>> and
>> > >> > >>>>> the last update was for the feature radar for the 1.15 release.
>> > It
>> > >> > >> has
>> > >> > >>>> been
>> > >> > >>>>> 2 years since the community discussed Flink's overall roadmap.
>> > >> > >>>>>
>> > >> > >>>>> I would like to raise two topics for discussion:
>> > >> > >>>>>
>> > >> > >>>>> 1. The new roadmap. This should be an updated version of the
>> > >> current
>> > >> > >>>>> roadmap[1].
>> > >> > >>>>> 2. A mechanism to regularly discuss and update the roadmap.
>> > >> > >>>>>
>> > >> > >>>>> To make the first topic discussion more efficient, Martijn and I
>> > >> > >>>> volunteer
>> > >> > >>>>> to summarize the ongoing big things of different components and
>> > >> > >>> present a
>> > >> > >>>>> roadmap draft to the community in the next few weeks. This
>> > should
>> > >> be
>> > >> > >> a
>> > >> > >>>> good
>> > >> > >>>>> starting point for a more detailed discussion.
>> > >> > >>>>>
>> > >> > >>>>> Regarding the regular update mechanism, there was a proposal in
>> > a
>> > >> > >>> thread
>> > >> > >>>>> [2] three years ago to make the release manager responsible for
>> > >> > >>> updating
>> > >> > >>>>> the roadmap. However, it appears that this was not documented
>> > as a
>> > >> > >>>> release
>> > >> > >>>>> management task [3], and the roadmap update wasn't performed for
>> > >> > >>> releases
>> > >> > >>>>> 1.16 and 1.17.
>> > >> > >>>>>
>> > >> > >>>>> In my opinion, making release managers responsible for keeping
>> > the
>> > >> > >>>> roadmap
>> > >> > >>>>> up to date is a good idea. Specifically, release managers of
>> > >> release
>> > >> > >> X
>> > >> > >>>> can
>> > >> > >>>>> kick off the roadmap update at the beginning of release X, which
>> > >> can
>> > >> > >>> be a
>> > >> > >>>>> joint task with collecting a feature list [4]. Additionally,
>> > >> release
>> > >> > >>>>> managers of release X-1 can help verify and remove the
>> > >> accomplished
>> > >> > >>> items
>> > >> > >>>>> from the roadmap and update the feature radar.
>> > >> > >>>>>
>> > >> > >>>>> What do you think? Do you have other ideas?
>> > >> > >>>>>
>> > >> > >>>>> Best,
>> > >> > >>>>> Jark & Martijn
>> > >> > >>>>>
>> > >> > >>>>> [1]: https://flink.apache.org/roadmap.html
>> > >> > >>>>> [2]:
>> > >> > >> https://lists.apache.org/thread/o0l3cg6yphxwrww0k7215jgtw3yfoybv
>> > >> > >>>>> [3]:
>> > >> > >>>>>
>> > >> > >>>>
>> > >> > >>>
>> > >> > >>
>> > >> >
>> > >>
>> > https://cwiki.apache.org/confluence/display/FLINK/Flink+Release+Management
>> > >> > >>>>> [4]:
>> > >> https://cwiki.apache.org/confluence/display/FLINK/1.18+Release
>> > >> > >>>>>
>> > >> > >>>>
>> > >> > >>>
>> > >> > >>
>> > >> >
>> > >> >
>> > >>
>> > >
>> >

Re: [DISCUSS] Update Flink Roadmap

[Shammon FY <zj...@gmail.com>]

Thanks @Jark for driving the Flink Roadmap.

As we discussed olap in the thread [1] and according to the suggestions
from @Xingtong Song, could we add a subsection in `Towards Streaming
Warehouses` or `Performance` that the short-lived query in Flink Session
Cluster is one of the future directions for Flink?

Best,
Shammon FY

On Mon, Aug 14, 2023 at 8:03 PM Jark Wu <im...@gmail.com> wrote:

> Thank you everyone for helping polish the roadmap [1].
>
> I think I have addressed all the comments and we have included all ongoing
> parts of Flink.
> Please feel free to take a last look. I'm going to prepare the pull request
> if there are no more concerns.
>
> Best,
> Jark
>
> [1]:
>
> https://docs.google.com/document/d/12BDiVKEsY-f7HI3suO_IxwzCmR04QcVqLarXgyJAb7c/edit
>
> On Sun, 13 Aug 2023 at 13:04, Yuan Mei <yu...@gmail.com> wrote:
>
> > Sorry for taking so long
> >
> > I've added a section about Flink Disaggregated State Management Evolution
> > in the attached doc.
> >
> > I found some of the contents might be overlapped with the "large-scale
> > streaming jobs". So that part might need some changes as well.
> >
> > Please let me know what you think.
> >
> > Best
> > Yuan
> >
> > On Mon, Jul 24, 2023 at 12:07 PM Yuan Mei <yu...@gmail.com>
> wrote:
> >
> > > Sorry have missed this email and respond a bit late.
> > >
> > > I will put a draft for the long-term vision for the state as well as
> > > large-scale state support into the roadmap.
> > >
> > > Best
> > > Yuan
> > >
> > > On Mon, Jul 17, 2023 at 10:34 AM Jark Wu <im...@gmail.com> wrote:
> > >
> > >> Hi Jiabao,
> > >>
> > >> Thank you for your suggestions. I have added them to the "Going
> Beyond a
> > >> SQL Stream/Batch Processing Engine" and "Large-Scale State Jobs"
> > sections.
> > >>
> > >> Best,
> > >> Jark
> > >>
> > >> On Thu, 13 Jul 2023 at 16:06, Jiabao Sun <jiabao.sun@xtransfer.cn
> > >> .invalid>
> > >> wrote:
> > >>
> > >> > Thanks Jark and Martijn for driving this.
> > >> >
> > >> > There are two suggestions about the Table API:
> > >> >
> > >> > - Add the JSON type to adapt to the no sql database type.
> > >> > - Remove changelog normalize operator for upsert stream.
> > >> >
> > >> >
> > >> > Best,
> > >> > Jiabao
> > >> >
> > >> >
> > >> > > 2023年7月13日 下午3:49，Jark Wu <im...@gmail.com> 写道：
> > >> > >
> > >> > > Hi all,
> > >> > >
> > >> > > Sorry for taking so long back here.
> > >> > >
> > >> > > Martijn and I have drafted the first version of the updated
> roadmap,
> > >> > > including the updated feature radar reflecting the current state
> of
> > >> > > different components.
> > >> > >
> > >> >
> > >>
> >
> https://docs.google.com/document/d/12BDiVKEsY-f7HI3suO_IxwzCmR04QcVqLarXgyJAb7c/edit
> > >> > >
> > >> > > Feel free to leave comments in the thread or the document.
> > >> > > We may miss mentioning something important, so your help in
> > enriching
> > >> > > the content is greatly appreciated.
> > >> > >
> > >> > > Best,
> > >> > > Jark & Martijn
> > >> > >
> > >> > >
> > >> > > On Fri, 2 Jun 2023 at 00:50, Jing Ge <ji...@ververica.com.invalid>
> > >> wrote:
> > >> > >
> > >> > >> Hi Jark,
> > >> > >>
> > >> > >> Fair enough. Let's do it like you suggested. Thanks!
> > >> > >>
> > >> > >> Best regards,
> > >> > >> Jing
> > >> > >>
> > >> > >> On Thu, Jun 1, 2023 at 6:00 PM Jark Wu <im...@gmail.com> wrote:
> > >> > >>
> > >> > >>> Hi Jing,
> > >> > >>>
> > >> > >>> This thread is for discussing the roadmap for versions 1.18,
> 2.0,
> > >> and
> > >> > >> even
> > >> > >>> more.
> > >> > >>> One of the outcomes of this discussion will be an updated
> version
> > of
> > >> > the
> > >> > >>> current roadmap.
> > >> > >>> Let's work together on refining the roadmap in this thread.
> > >> > >>>
> > >> > >>> Best,
> > >> > >>> Jark
> > >> > >>>
> > >> > >>> On Thu, 1 Jun 2023 at 23:25, Jing Ge <jing@ververica.com.invalid
> >
> > >> > wrote:
> > >> > >>>
> > >> > >>>> Hi Jark,
> > >> > >>>>
> > >> > >>>> Thanks for driving it! For point 2, since we are developing
> 1.18
> > >> now,
> > >> > >>>> does it make sense to update the roadmap this time while we are
> > >> > >> releasing
> > >> > >>>> 1.18? This discussion thread will be focusing on the Flink 2.0
> > >> > roadmap,
> > >> > >>> as
> > >> > >>>> you mentioned previously. WDYT?
> > >> > >>>>
> > >> > >>>> Best regards,
> > >> > >>>> Jing
> > >> > >>>>
> > >> > >>>> On Thu, Jun 1, 2023 at 3:31 PM Jark Wu <im...@gmail.com>
> wrote:
> > >> > >>>>
> > >> > >>>>> Hi all,
> > >> > >>>>>
> > >> > >>>>> Martijn and I would like to initiate a discussion on the Flink
> > >> > >> roadmap,
> > >> > >>>>> which should cover the project's long-term roadmap and the
> > regular
> > >> > >>> update
> > >> > >>>>> mechanism.
> > >> > >>>>>
> > >> > >>>>> Xintong has already started a discussion about Flink 2.0
> > planning.
> > >> > >> One
> > >> > >>> of
> > >> > >>>>> the points raised in that discussion is that we should have a
> > >> > >>> high-level
> > >> > >>>>> discussion of the roadmap to present where the project is
> > heading
> > >> > >>> (which
> > >> > >>>>> doesn't necessarily need to block the Flink 2.0 planning).
> > >> Moreover,
> > >> > >>> the
> > >> > >>>>> roadmap on the Flink website [1] hasn't been updated for half
> a
> > >> year,
> > >> > >>> and
> > >> > >>>>> the last update was for the feature radar for the 1.15
> release.
> > It
> > >> > >> has
> > >> > >>>> been
> > >> > >>>>> 2 years since the community discussed Flink's overall roadmap.
> > >> > >>>>>
> > >> > >>>>> I would like to raise two topics for discussion:
> > >> > >>>>>
> > >> > >>>>> 1. The new roadmap. This should be an updated version of the
> > >> current
> > >> > >>>>> roadmap[1].
> > >> > >>>>> 2. A mechanism to regularly discuss and update the roadmap.
> > >> > >>>>>
> > >> > >>>>> To make the first topic discussion more efficient, Martijn
> and I
> > >> > >>>> volunteer
> > >> > >>>>> to summarize the ongoing big things of different components
> and
> > >> > >>> present a
> > >> > >>>>> roadmap draft to the community in the next few weeks. This
> > should
> > >> be
> > >> > >> a
> > >> > >>>> good
> > >> > >>>>> starting point for a more detailed discussion.
> > >> > >>>>>
> > >> > >>>>> Regarding the regular update mechanism, there was a proposal
> in
> > a
> > >> > >>> thread
> > >> > >>>>> [2] three years ago to make the release manager responsible
> for
> > >> > >>> updating
> > >> > >>>>> the roadmap. However, it appears that this was not documented
> > as a
> > >> > >>>> release
> > >> > >>>>> management task [3], and the roadmap update wasn't performed
> for
> > >> > >>> releases
> > >> > >>>>> 1.16 and 1.17.
> > >> > >>>>>
> > >> > >>>>> In my opinion, making release managers responsible for keeping
> > the
> > >> > >>>> roadmap
> > >> > >>>>> up to date is a good idea. Specifically, release managers of
> > >> release
> > >> > >> X
> > >> > >>>> can
> > >> > >>>>> kick off the roadmap update at the beginning of release X,
> which
> > >> can
> > >> > >>> be a
> > >> > >>>>> joint task with collecting a feature list [4]. Additionally,
> > >> release
> > >> > >>>>> managers of release X-1 can help verify and remove the
> > >> accomplished
> > >> > >>> items
> > >> > >>>>> from the roadmap and update the feature radar.
> > >> > >>>>>
> > >> > >>>>> What do you think? Do you have other ideas?
> > >> > >>>>>
> > >> > >>>>> Best,
> > >> > >>>>> Jark & Martijn
> > >> > >>>>>
> > >> > >>>>> [1]: https://flink.apache.org/roadmap.html
> > >> > >>>>> [2]:
> > >> > >> https://lists.apache.org/thread/o0l3cg6yphxwrww0k7215jgtw3yfoybv
> > >> > >>>>> [3]:
> > >> > >>>>>
> > >> > >>>>
> > >> > >>>
> > >> > >>
> > >> >
> > >>
> >
> https://cwiki.apache.org/confluence/display/FLINK/Flink+Release+Management
> > >> > >>>>> [4]:
> > >> https://cwiki.apache.org/confluence/display/FLINK/1.18+Release
> > >> > >>>>>
> > >> > >>>>
> > >> > >>>
> > >> > >>
> > >> >
> > >> >
> > >>
> > >
> >
>

Re: [DISCUSS] Update Flink Roadmap

[Jark Wu <im...@gmail.com>]

Thank you everyone for helping polish the roadmap [1].

I think I have addressed all the comments and we have included all ongoing
parts of Flink.
Please feel free to take a last look. I'm going to prepare the pull request
if there are no more concerns.

Best,
Jark

[1]:
https://docs.google.com/document/d/12BDiVKEsY-f7HI3suO_IxwzCmR04QcVqLarXgyJAb7c/edit

On Sun, 13 Aug 2023 at 13:04, Yuan Mei <yu...@gmail.com> wrote:

> Sorry for taking so long
>
> I've added a section about Flink Disaggregated State Management Evolution
> in the attached doc.
>
> I found some of the contents might be overlapped with the "large-scale
> streaming jobs". So that part might need some changes as well.
>
> Please let me know what you think.
>
> Best
> Yuan
>
> On Mon, Jul 24, 2023 at 12:07 PM Yuan Mei <yu...@gmail.com> wrote:
>
> > Sorry have missed this email and respond a bit late.
> >
> > I will put a draft for the long-term vision for the state as well as
> > large-scale state support into the roadmap.
> >
> > Best
> > Yuan
> >
> > On Mon, Jul 17, 2023 at 10:34 AM Jark Wu <im...@gmail.com> wrote:
> >
> >> Hi Jiabao,
> >>
> >> Thank you for your suggestions. I have added them to the "Going Beyond a
> >> SQL Stream/Batch Processing Engine" and "Large-Scale State Jobs"
> sections.
> >>
> >> Best,
> >> Jark
> >>
> >> On Thu, 13 Jul 2023 at 16:06, Jiabao Sun <jiabao.sun@xtransfer.cn
> >> .invalid>
> >> wrote:
> >>
> >> > Thanks Jark and Martijn for driving this.
> >> >
> >> > There are two suggestions about the Table API:
> >> >
> >> > - Add the JSON type to adapt to the no sql database type.
> >> > - Remove changelog normalize operator for upsert stream.
> >> >
> >> >
> >> > Best,
> >> > Jiabao
> >> >
> >> >
> >> > > 2023年7月13日 下午3:49，Jark Wu <im...@gmail.com> 写道：
> >> > >
> >> > > Hi all,
> >> > >
> >> > > Sorry for taking so long back here.
> >> > >
> >> > > Martijn and I have drafted the first version of the updated roadmap,
> >> > > including the updated feature radar reflecting the current state of
> >> > > different components.
> >> > >
> >> >
> >>
> https://docs.google.com/document/d/12BDiVKEsY-f7HI3suO_IxwzCmR04QcVqLarXgyJAb7c/edit
> >> > >
> >> > > Feel free to leave comments in the thread or the document.
> >> > > We may miss mentioning something important, so your help in
> enriching
> >> > > the content is greatly appreciated.
> >> > >
> >> > > Best,
> >> > > Jark & Martijn
> >> > >
> >> > >
> >> > > On Fri, 2 Jun 2023 at 00:50, Jing Ge <ji...@ververica.com.invalid>
> >> wrote:
> >> > >
> >> > >> Hi Jark,
> >> > >>
> >> > >> Fair enough. Let's do it like you suggested. Thanks!
> >> > >>
> >> > >> Best regards,
> >> > >> Jing
> >> > >>
> >> > >> On Thu, Jun 1, 2023 at 6:00 PM Jark Wu <im...@gmail.com> wrote:
> >> > >>
> >> > >>> Hi Jing,
> >> > >>>
> >> > >>> This thread is for discussing the roadmap for versions 1.18, 2.0,
> >> and
> >> > >> even
> >> > >>> more.
> >> > >>> One of the outcomes of this discussion will be an updated version
> of
> >> > the
> >> > >>> current roadmap.
> >> > >>> Let's work together on refining the roadmap in this thread.
> >> > >>>
> >> > >>> Best,
> >> > >>> Jark
> >> > >>>
> >> > >>> On Thu, 1 Jun 2023 at 23:25, Jing Ge <ji...@ververica.com.invalid>
> >> > wrote:
> >> > >>>
> >> > >>>> Hi Jark,
> >> > >>>>
> >> > >>>> Thanks for driving it! For point 2, since we are developing 1.18
> >> now,
> >> > >>>> does it make sense to update the roadmap this time while we are
> >> > >> releasing
> >> > >>>> 1.18? This discussion thread will be focusing on the Flink 2.0
> >> > roadmap,
> >> > >>> as
> >> > >>>> you mentioned previously. WDYT?
> >> > >>>>
> >> > >>>> Best regards,
> >> > >>>> Jing
> >> > >>>>
> >> > >>>> On Thu, Jun 1, 2023 at 3:31 PM Jark Wu <im...@gmail.com> wrote:
> >> > >>>>
> >> > >>>>> Hi all,
> >> > >>>>>
> >> > >>>>> Martijn and I would like to initiate a discussion on the Flink
> >> > >> roadmap,
> >> > >>>>> which should cover the project's long-term roadmap and the
> regular
> >> > >>> update
> >> > >>>>> mechanism.
> >> > >>>>>
> >> > >>>>> Xintong has already started a discussion about Flink 2.0
> planning.
> >> > >> One
> >> > >>> of
> >> > >>>>> the points raised in that discussion is that we should have a
> >> > >>> high-level
> >> > >>>>> discussion of the roadmap to present where the project is
> heading
> >> > >>> (which
> >> > >>>>> doesn't necessarily need to block the Flink 2.0 planning).
> >> Moreover,
> >> > >>> the
> >> > >>>>> roadmap on the Flink website [1] hasn't been updated for half a
> >> year,
> >> > >>> and
> >> > >>>>> the last update was for the feature radar for the 1.15 release.
> It
> >> > >> has
> >> > >>>> been
> >> > >>>>> 2 years since the community discussed Flink's overall roadmap.
> >> > >>>>>
> >> > >>>>> I would like to raise two topics for discussion:
> >> > >>>>>
> >> > >>>>> 1. The new roadmap. This should be an updated version of the
> >> current
> >> > >>>>> roadmap[1].
> >> > >>>>> 2. A mechanism to regularly discuss and update the roadmap.
> >> > >>>>>
> >> > >>>>> To make the first topic discussion more efficient, Martijn and I
> >> > >>>> volunteer
> >> > >>>>> to summarize the ongoing big things of different components and
> >> > >>> present a
> >> > >>>>> roadmap draft to the community in the next few weeks. This
> should
> >> be
> >> > >> a
> >> > >>>> good
> >> > >>>>> starting point for a more detailed discussion.
> >> > >>>>>
> >> > >>>>> Regarding the regular update mechanism, there was a proposal in
> a
> >> > >>> thread
> >> > >>>>> [2] three years ago to make the release manager responsible for
> >> > >>> updating
> >> > >>>>> the roadmap. However, it appears that this was not documented
> as a
> >> > >>>> release
> >> > >>>>> management task [3], and the roadmap update wasn't performed for
> >> > >>> releases
> >> > >>>>> 1.16 and 1.17.
> >> > >>>>>
> >> > >>>>> In my opinion, making release managers responsible for keeping
> the
> >> > >>>> roadmap
> >> > >>>>> up to date is a good idea. Specifically, release managers of
> >> release
> >> > >> X
> >> > >>>> can
> >> > >>>>> kick off the roadmap update at the beginning of release X, which
> >> can
> >> > >>> be a
> >> > >>>>> joint task with collecting a feature list [4]. Additionally,
> >> release
> >> > >>>>> managers of release X-1 can help verify and remove the
> >> accomplished
> >> > >>> items
> >> > >>>>> from the roadmap and update the feature radar.
> >> > >>>>>
> >> > >>>>> What do you think? Do you have other ideas?
> >> > >>>>>
> >> > >>>>> Best,
> >> > >>>>> Jark & Martijn
> >> > >>>>>
> >> > >>>>> [1]: https://flink.apache.org/roadmap.html
> >> > >>>>> [2]:
> >> > >> https://lists.apache.org/thread/o0l3cg6yphxwrww0k7215jgtw3yfoybv
> >> > >>>>> [3]:
> >> > >>>>>
> >> > >>>>
> >> > >>>
> >> > >>
> >> >
> >>
> https://cwiki.apache.org/confluence/display/FLINK/Flink+Release+Management
> >> > >>>>> [4]:
> >> https://cwiki.apache.org/confluence/display/FLINK/1.18+Release
> >> > >>>>>
> >> > >>>>
> >> > >>>
> >> > >>
> >> >
> >> >
> >>
> >
>

Re: [DISCUSS] Update Flink Roadmap

[Yuan Mei <yu...@gmail.com>]

Sorry for taking so long

I've added a section about Flink Disaggregated State Management Evolution
in the attached doc.

I found some of the contents might be overlapped with the "large-scale
streaming jobs". So that part might need some changes as well.

Please let me know what you think.

Best
Yuan

On Mon, Jul 24, 2023 at 12:07 PM Yuan Mei <yu...@gmail.com> wrote:

> Sorry have missed this email and respond a bit late.
>
> I will put a draft for the long-term vision for the state as well as
> large-scale state support into the roadmap.
>
> Best
> Yuan
>
> On Mon, Jul 17, 2023 at 10:34 AM Jark Wu <im...@gmail.com> wrote:
>
>> Hi Jiabao,
>>
>> Thank you for your suggestions. I have added them to the "Going Beyond a
>> SQL Stream/Batch Processing Engine" and "Large-Scale State Jobs" sections.
>>
>> Best,
>> Jark
>>
>> On Thu, 13 Jul 2023 at 16:06, Jiabao Sun <jiabao.sun@xtransfer.cn
>> .invalid>
>> wrote:
>>
>> > Thanks Jark and Martijn for driving this.
>> >
>> > There are two suggestions about the Table API:
>> >
>> > - Add the JSON type to adapt to the no sql database type.
>> > - Remove changelog normalize operator for upsert stream.
>> >
>> >
>> > Best,
>> > Jiabao
>> >
>> >
>> > > 2023年7月13日 下午3:49，Jark Wu <im...@gmail.com> 写道：
>> > >
>> > > Hi all,
>> > >
>> > > Sorry for taking so long back here.
>> > >
>> > > Martijn and I have drafted the first version of the updated roadmap,
>> > > including the updated feature radar reflecting the current state of
>> > > different components.
>> > >
>> >
>> https://docs.google.com/document/d/12BDiVKEsY-f7HI3suO_IxwzCmR04QcVqLarXgyJAb7c/edit
>> > >
>> > > Feel free to leave comments in the thread or the document.
>> > > We may miss mentioning something important, so your help in enriching
>> > > the content is greatly appreciated.
>> > >
>> > > Best,
>> > > Jark & Martijn
>> > >
>> > >
>> > > On Fri, 2 Jun 2023 at 00:50, Jing Ge <ji...@ververica.com.invalid>
>> wrote:
>> > >
>> > >> Hi Jark,
>> > >>
>> > >> Fair enough. Let's do it like you suggested. Thanks!
>> > >>
>> > >> Best regards,
>> > >> Jing
>> > >>
>> > >> On Thu, Jun 1, 2023 at 6:00 PM Jark Wu <im...@gmail.com> wrote:
>> > >>
>> > >>> Hi Jing,
>> > >>>
>> > >>> This thread is for discussing the roadmap for versions 1.18, 2.0,
>> and
>> > >> even
>> > >>> more.
>> > >>> One of the outcomes of this discussion will be an updated version of
>> > the
>> > >>> current roadmap.
>> > >>> Let's work together on refining the roadmap in this thread.
>> > >>>
>> > >>> Best,
>> > >>> Jark
>> > >>>
>> > >>> On Thu, 1 Jun 2023 at 23:25, Jing Ge <ji...@ververica.com.invalid>
>> > wrote:
>> > >>>
>> > >>>> Hi Jark,
>> > >>>>
>> > >>>> Thanks for driving it! For point 2, since we are developing 1.18
>> now,
>> > >>>> does it make sense to update the roadmap this time while we are
>> > >> releasing
>> > >>>> 1.18? This discussion thread will be focusing on the Flink 2.0
>> > roadmap,
>> > >>> as
>> > >>>> you mentioned previously. WDYT?
>> > >>>>
>> > >>>> Best regards,
>> > >>>> Jing
>> > >>>>
>> > >>>> On Thu, Jun 1, 2023 at 3:31 PM Jark Wu <im...@gmail.com> wrote:
>> > >>>>
>> > >>>>> Hi all,
>> > >>>>>
>> > >>>>> Martijn and I would like to initiate a discussion on the Flink
>> > >> roadmap,
>> > >>>>> which should cover the project's long-term roadmap and the regular
>> > >>> update
>> > >>>>> mechanism.
>> > >>>>>
>> > >>>>> Xintong has already started a discussion about Flink 2.0 planning.
>> > >> One
>> > >>> of
>> > >>>>> the points raised in that discussion is that we should have a
>> > >>> high-level
>> > >>>>> discussion of the roadmap to present where the project is heading
>> > >>> (which
>> > >>>>> doesn't necessarily need to block the Flink 2.0 planning).
>> Moreover,
>> > >>> the
>> > >>>>> roadmap on the Flink website [1] hasn't been updated for half a
>> year,
>> > >>> and
>> > >>>>> the last update was for the feature radar for the 1.15 release. It
>> > >> has
>> > >>>> been
>> > >>>>> 2 years since the community discussed Flink's overall roadmap.
>> > >>>>>
>> > >>>>> I would like to raise two topics for discussion:
>> > >>>>>
>> > >>>>> 1. The new roadmap. This should be an updated version of the
>> current
>> > >>>>> roadmap[1].
>> > >>>>> 2. A mechanism to regularly discuss and update the roadmap.
>> > >>>>>
>> > >>>>> To make the first topic discussion more efficient, Martijn and I
>> > >>>> volunteer
>> > >>>>> to summarize the ongoing big things of different components and
>> > >>> present a
>> > >>>>> roadmap draft to the community in the next few weeks. This should
>> be
>> > >> a
>> > >>>> good
>> > >>>>> starting point for a more detailed discussion.
>> > >>>>>
>> > >>>>> Regarding the regular update mechanism, there was a proposal in a
>> > >>> thread
>> > >>>>> [2] three years ago to make the release manager responsible for
>> > >>> updating
>> > >>>>> the roadmap. However, it appears that this was not documented as a
>> > >>>> release
>> > >>>>> management task [3], and the roadmap update wasn't performed for
>> > >>> releases
>> > >>>>> 1.16 and 1.17.
>> > >>>>>
>> > >>>>> In my opinion, making release managers responsible for keeping the
>> > >>>> roadmap
>> > >>>>> up to date is a good idea. Specifically, release managers of
>> release
>> > >> X
>> > >>>> can
>> > >>>>> kick off the roadmap update at the beginning of release X, which
>> can
>> > >>> be a
>> > >>>>> joint task with collecting a feature list [4]. Additionally,
>> release
>> > >>>>> managers of release X-1 can help verify and remove the
>> accomplished
>> > >>> items
>> > >>>>> from the roadmap and update the feature radar.
>> > >>>>>
>> > >>>>> What do you think? Do you have other ideas?
>> > >>>>>
>> > >>>>> Best,
>> > >>>>> Jark & Martijn
>> > >>>>>
>> > >>>>> [1]: https://flink.apache.org/roadmap.html
>> > >>>>> [2]:
>> > >> https://lists.apache.org/thread/o0l3cg6yphxwrww0k7215jgtw3yfoybv
>> > >>>>> [3]:
>> > >>>>>
>> > >>>>
>> > >>>
>> > >>
>> >
>> https://cwiki.apache.org/confluence/display/FLINK/Flink+Release+Management
>> > >>>>> [4]:
>> https://cwiki.apache.org/confluence/display/FLINK/1.18+Release
>> > >>>>>
>> > >>>>
>> > >>>
>> > >>
>> >
>> >
>>
>

Re: [DISCUSS] Update Flink Roadmap

[Yuan Mei <yu...@gmail.com>]

Sorry have missed this email and respond a bit late.

I will put a draft for the long-term vision for the state as well as
large-scale state support into the roadmap.

Best
Yuan

On Mon, Jul 17, 2023 at 10:34 AM Jark Wu <im...@gmail.com> wrote:

> Hi Jiabao,
>
> Thank you for your suggestions. I have added them to the "Going Beyond a
> SQL Stream/Batch Processing Engine" and "Large-Scale State Jobs" sections.
>
> Best,
> Jark
>
> On Thu, 13 Jul 2023 at 16:06, Jiabao Sun <ji...@xtransfer.cn.invalid>
> wrote:
>
> > Thanks Jark and Martijn for driving this.
> >
> > There are two suggestions about the Table API:
> >
> > - Add the JSON type to adapt to the no sql database type.
> > - Remove changelog normalize operator for upsert stream.
> >
> >
> > Best,
> > Jiabao
> >
> >
> > > 2023年7月13日 下午3:49，Jark Wu <im...@gmail.com> 写道：
> > >
> > > Hi all,
> > >
> > > Sorry for taking so long back here.
> > >
> > > Martijn and I have drafted the first version of the updated roadmap,
> > > including the updated feature radar reflecting the current state of
> > > different components.
> > >
> >
> https://docs.google.com/document/d/12BDiVKEsY-f7HI3suO_IxwzCmR04QcVqLarXgyJAb7c/edit
> > >
> > > Feel free to leave comments in the thread or the document.
> > > We may miss mentioning something important, so your help in enriching
> > > the content is greatly appreciated.
> > >
> > > Best,
> > > Jark & Martijn
> > >
> > >
> > > On Fri, 2 Jun 2023 at 00:50, Jing Ge <ji...@ververica.com.invalid>
> wrote:
> > >
> > >> Hi Jark,
> > >>
> > >> Fair enough. Let's do it like you suggested. Thanks!
> > >>
> > >> Best regards,
> > >> Jing
> > >>
> > >> On Thu, Jun 1, 2023 at 6:00 PM Jark Wu <im...@gmail.com> wrote:
> > >>
> > >>> Hi Jing,
> > >>>
> > >>> This thread is for discussing the roadmap for versions 1.18, 2.0, and
> > >> even
> > >>> more.
> > >>> One of the outcomes of this discussion will be an updated version of
> > the
> > >>> current roadmap.
> > >>> Let's work together on refining the roadmap in this thread.
> > >>>
> > >>> Best,
> > >>> Jark
> > >>>
> > >>> On Thu, 1 Jun 2023 at 23:25, Jing Ge <ji...@ververica.com.invalid>
> > wrote:
> > >>>
> > >>>> Hi Jark,
> > >>>>
> > >>>> Thanks for driving it! For point 2, since we are developing 1.18
> now,
> > >>>> does it make sense to update the roadmap this time while we are
> > >> releasing
> > >>>> 1.18? This discussion thread will be focusing on the Flink 2.0
> > roadmap,
> > >>> as
> > >>>> you mentioned previously. WDYT?
> > >>>>
> > >>>> Best regards,
> > >>>> Jing
> > >>>>
> > >>>> On Thu, Jun 1, 2023 at 3:31 PM Jark Wu <im...@gmail.com> wrote:
> > >>>>
> > >>>>> Hi all,
> > >>>>>
> > >>>>> Martijn and I would like to initiate a discussion on the Flink
> > >> roadmap,
> > >>>>> which should cover the project's long-term roadmap and the regular
> > >>> update
> > >>>>> mechanism.
> > >>>>>
> > >>>>> Xintong has already started a discussion about Flink 2.0 planning.
> > >> One
> > >>> of
> > >>>>> the points raised in that discussion is that we should have a
> > >>> high-level
> > >>>>> discussion of the roadmap to present where the project is heading
> > >>> (which
> > >>>>> doesn't necessarily need to block the Flink 2.0 planning).
> Moreover,
> > >>> the
> > >>>>> roadmap on the Flink website [1] hasn't been updated for half a
> year,
> > >>> and
> > >>>>> the last update was for the feature radar for the 1.15 release. It
> > >> has
> > >>>> been
> > >>>>> 2 years since the community discussed Flink's overall roadmap.
> > >>>>>
> > >>>>> I would like to raise two topics for discussion:
> > >>>>>
> > >>>>> 1. The new roadmap. This should be an updated version of the
> current
> > >>>>> roadmap[1].
> > >>>>> 2. A mechanism to regularly discuss and update the roadmap.
> > >>>>>
> > >>>>> To make the first topic discussion more efficient, Martijn and I
> > >>>> volunteer
> > >>>>> to summarize the ongoing big things of different components and
> > >>> present a
> > >>>>> roadmap draft to the community in the next few weeks. This should
> be
> > >> a
> > >>>> good
> > >>>>> starting point for a more detailed discussion.
> > >>>>>
> > >>>>> Regarding the regular update mechanism, there was a proposal in a
> > >>> thread
> > >>>>> [2] three years ago to make the release manager responsible for
> > >>> updating
> > >>>>> the roadmap. However, it appears that this was not documented as a
> > >>>> release
> > >>>>> management task [3], and the roadmap update wasn't performed for
> > >>> releases
> > >>>>> 1.16 and 1.17.
> > >>>>>
> > >>>>> In my opinion, making release managers responsible for keeping the
> > >>>> roadmap
> > >>>>> up to date is a good idea. Specifically, release managers of
> release
> > >> X
> > >>>> can
> > >>>>> kick off the roadmap update at the beginning of release X, which
> can
> > >>> be a
> > >>>>> joint task with collecting a feature list [4]. Additionally,
> release
> > >>>>> managers of release X-1 can help verify and remove the accomplished
> > >>> items
> > >>>>> from the roadmap and update the feature radar.
> > >>>>>
> > >>>>> What do you think? Do you have other ideas?
> > >>>>>
> > >>>>> Best,
> > >>>>> Jark & Martijn
> > >>>>>
> > >>>>> [1]: https://flink.apache.org/roadmap.html
> > >>>>> [2]:
> > >> https://lists.apache.org/thread/o0l3cg6yphxwrww0k7215jgtw3yfoybv
> > >>>>> [3]:
> > >>>>>
> > >>>>
> > >>>
> > >>
> >
> https://cwiki.apache.org/confluence/display/FLINK/Flink+Release+Management
> > >>>>> [4]:
> https://cwiki.apache.org/confluence/display/FLINK/1.18+Release
> > >>>>>
> > >>>>
> > >>>
> > >>
> >
> >
>

Re: [DISCUSS] Update Flink Roadmap

[Jark Wu <im...@gmail.com>]

Hi Jiabao,

Thank you for your suggestions. I have added them to the "Going Beyond a
SQL Stream/Batch Processing Engine" and "Large-Scale State Jobs" sections.

Best,
Jark

On Thu, 13 Jul 2023 at 16:06, Jiabao Sun <ji...@xtransfer.cn.invalid>
wrote:

> Thanks Jark and Martijn for driving this.
>
> There are two suggestions about the Table API:
>
> - Add the JSON type to adapt to the no sql database type.
> - Remove changelog normalize operator for upsert stream.
>
>
> Best,
> Jiabao
>
>
> > 2023年7月13日 下午3:49，Jark Wu <im...@gmail.com> 写道：
> >
> > Hi all,
> >
> > Sorry for taking so long back here.
> >
> > Martijn and I have drafted the first version of the updated roadmap,
> > including the updated feature radar reflecting the current state of
> > different components.
> >
> https://docs.google.com/document/d/12BDiVKEsY-f7HI3suO_IxwzCmR04QcVqLarXgyJAb7c/edit
> >
> > Feel free to leave comments in the thread or the document.
> > We may miss mentioning something important, so your help in enriching
> > the content is greatly appreciated.
> >
> > Best,
> > Jark & Martijn
> >
> >
> > On Fri, 2 Jun 2023 at 00:50, Jing Ge <ji...@ververica.com.invalid> wrote:
> >
> >> Hi Jark,
> >>
> >> Fair enough. Let's do it like you suggested. Thanks!
> >>
> >> Best regards,
> >> Jing
> >>
> >> On Thu, Jun 1, 2023 at 6:00 PM Jark Wu <im...@gmail.com> wrote:
> >>
> >>> Hi Jing,
> >>>
> >>> This thread is for discussing the roadmap for versions 1.18, 2.0, and
> >> even
> >>> more.
> >>> One of the outcomes of this discussion will be an updated version of
> the
> >>> current roadmap.
> >>> Let's work together on refining the roadmap in this thread.
> >>>
> >>> Best,
> >>> Jark
> >>>
> >>> On Thu, 1 Jun 2023 at 23:25, Jing Ge <ji...@ververica.com.invalid>
> wrote:
> >>>
> >>>> Hi Jark,
> >>>>
> >>>> Thanks for driving it! For point 2, since we are developing 1.18 now,
> >>>> does it make sense to update the roadmap this time while we are
> >> releasing
> >>>> 1.18? This discussion thread will be focusing on the Flink 2.0
> roadmap,
> >>> as
> >>>> you mentioned previously. WDYT?
> >>>>
> >>>> Best regards,
> >>>> Jing
> >>>>
> >>>> On Thu, Jun 1, 2023 at 3:31 PM Jark Wu <im...@gmail.com> wrote:
> >>>>
> >>>>> Hi all,
> >>>>>
> >>>>> Martijn and I would like to initiate a discussion on the Flink
> >> roadmap,
> >>>>> which should cover the project's long-term roadmap and the regular
> >>> update
> >>>>> mechanism.
> >>>>>
> >>>>> Xintong has already started a discussion about Flink 2.0 planning.
> >> One
> >>> of
> >>>>> the points raised in that discussion is that we should have a
> >>> high-level
> >>>>> discussion of the roadmap to present where the project is heading
> >>> (which
> >>>>> doesn't necessarily need to block the Flink 2.0 planning). Moreover,
> >>> the
> >>>>> roadmap on the Flink website [1] hasn't been updated for half a year,
> >>> and
> >>>>> the last update was for the feature radar for the 1.15 release. It
> >> has
> >>>> been
> >>>>> 2 years since the community discussed Flink's overall roadmap.
> >>>>>
> >>>>> I would like to raise two topics for discussion:
> >>>>>
> >>>>> 1. The new roadmap. This should be an updated version of the current
> >>>>> roadmap[1].
> >>>>> 2. A mechanism to regularly discuss and update the roadmap.
> >>>>>
> >>>>> To make the first topic discussion more efficient, Martijn and I
> >>>> volunteer
> >>>>> to summarize the ongoing big things of different components and
> >>> present a
> >>>>> roadmap draft to the community in the next few weeks. This should be
> >> a
> >>>> good
> >>>>> starting point for a more detailed discussion.
> >>>>>
> >>>>> Regarding the regular update mechanism, there was a proposal in a
> >>> thread
> >>>>> [2] three years ago to make the release manager responsible for
> >>> updating
> >>>>> the roadmap. However, it appears that this was not documented as a
> >>>> release
> >>>>> management task [3], and the roadmap update wasn't performed for
> >>> releases
> >>>>> 1.16 and 1.17.
> >>>>>
> >>>>> In my opinion, making release managers responsible for keeping the
> >>>> roadmap
> >>>>> up to date is a good idea. Specifically, release managers of release
> >> X
> >>>> can
> >>>>> kick off the roadmap update at the beginning of release X, which can
> >>> be a
> >>>>> joint task with collecting a feature list [4]. Additionally, release
> >>>>> managers of release X-1 can help verify and remove the accomplished
> >>> items
> >>>>> from the roadmap and update the feature radar.
> >>>>>
> >>>>> What do you think? Do you have other ideas?
> >>>>>
> >>>>> Best,
> >>>>> Jark & Martijn
> >>>>>
> >>>>> [1]: https://flink.apache.org/roadmap.html
> >>>>> [2]:
> >> https://lists.apache.org/thread/o0l3cg6yphxwrww0k7215jgtw3yfoybv
> >>>>> [3]:
> >>>>>
> >>>>
> >>>
> >>
> https://cwiki.apache.org/confluence/display/FLINK/Flink+Release+Management
> >>>>> [4]: https://cwiki.apache.org/confluence/display/FLINK/1.18+Release
> >>>>>
> >>>>
> >>>
> >>
>
>

Re: [DISCUSS] Update Flink Roadmap

[Jiabao Sun <ji...@xtransfer.cn.INVALID>]

Thanks Jark and Martijn for driving this.

There are two suggestions about the Table API:

- Add the JSON type to adapt to the no sql database type.
- Remove changelog normalize operator for upsert stream.


Best,
Jiabao


> 2023年7月13日 下午3:49，Jark Wu <im...@gmail.com> 写道：
> 
> Hi all,
> 
> Sorry for taking so long back here.
> 
> Martijn and I have drafted the first version of the updated roadmap,
> including the updated feature radar reflecting the current state of
> different components.
> https://docs.google.com/document/d/12BDiVKEsY-f7HI3suO_IxwzCmR04QcVqLarXgyJAb7c/edit
> 
> Feel free to leave comments in the thread or the document.
> We may miss mentioning something important, so your help in enriching
> the content is greatly appreciated.
> 
> Best,
> Jark & Martijn
> 
> 
> On Fri, 2 Jun 2023 at 00:50, Jing Ge <ji...@ververica.com.invalid> wrote:
> 
>> Hi Jark,
>> 
>> Fair enough. Let's do it like you suggested. Thanks!
>> 
>> Best regards,
>> Jing
>> 
>> On Thu, Jun 1, 2023 at 6:00 PM Jark Wu <im...@gmail.com> wrote:
>> 
>>> Hi Jing,
>>> 
>>> This thread is for discussing the roadmap for versions 1.18, 2.0, and
>> even
>>> more.
>>> One of the outcomes of this discussion will be an updated version of the
>>> current roadmap.
>>> Let's work together on refining the roadmap in this thread.
>>> 
>>> Best,
>>> Jark
>>> 
>>> On Thu, 1 Jun 2023 at 23:25, Jing Ge <ji...@ververica.com.invalid> wrote:
>>> 
>>>> Hi Jark,
>>>> 
>>>> Thanks for driving it! For point 2, since we are developing 1.18 now,
>>>> does it make sense to update the roadmap this time while we are
>> releasing
>>>> 1.18? This discussion thread will be focusing on the Flink 2.0 roadmap,
>>> as
>>>> you mentioned previously. WDYT?
>>>> 
>>>> Best regards,
>>>> Jing
>>>> 
>>>> On Thu, Jun 1, 2023 at 3:31 PM Jark Wu <im...@gmail.com> wrote:
>>>> 
>>>>> Hi all,
>>>>> 
>>>>> Martijn and I would like to initiate a discussion on the Flink
>> roadmap,
>>>>> which should cover the project's long-term roadmap and the regular
>>> update
>>>>> mechanism.
>>>>> 
>>>>> Xintong has already started a discussion about Flink 2.0 planning.
>> One
>>> of
>>>>> the points raised in that discussion is that we should have a
>>> high-level
>>>>> discussion of the roadmap to present where the project is heading
>>> (which
>>>>> doesn't necessarily need to block the Flink 2.0 planning). Moreover,
>>> the
>>>>> roadmap on the Flink website [1] hasn't been updated for half a year,
>>> and
>>>>> the last update was for the feature radar for the 1.15 release. It
>> has
>>>> been
>>>>> 2 years since the community discussed Flink's overall roadmap.
>>>>> 
>>>>> I would like to raise two topics for discussion:
>>>>> 
>>>>> 1. The new roadmap. This should be an updated version of the current
>>>>> roadmap[1].
>>>>> 2. A mechanism to regularly discuss and update the roadmap.
>>>>> 
>>>>> To make the first topic discussion more efficient, Martijn and I
>>>> volunteer
>>>>> to summarize the ongoing big things of different components and
>>> present a
>>>>> roadmap draft to the community in the next few weeks. This should be
>> a
>>>> good
>>>>> starting point for a more detailed discussion.
>>>>> 
>>>>> Regarding the regular update mechanism, there was a proposal in a
>>> thread
>>>>> [2] three years ago to make the release manager responsible for
>>> updating
>>>>> the roadmap. However, it appears that this was not documented as a
>>>> release
>>>>> management task [3], and the roadmap update wasn't performed for
>>> releases
>>>>> 1.16 and 1.17.
>>>>> 
>>>>> In my opinion, making release managers responsible for keeping the
>>>> roadmap
>>>>> up to date is a good idea. Specifically, release managers of release
>> X
>>>> can
>>>>> kick off the roadmap update at the beginning of release X, which can
>>> be a
>>>>> joint task with collecting a feature list [4]. Additionally, release
>>>>> managers of release X-1 can help verify and remove the accomplished
>>> items
>>>>> from the roadmap and update the feature radar.
>>>>> 
>>>>> What do you think? Do you have other ideas?
>>>>> 
>>>>> Best,
>>>>> Jark & Martijn
>>>>> 
>>>>> [1]: https://flink.apache.org/roadmap.html
>>>>> [2]:
>> https://lists.apache.org/thread/o0l3cg6yphxwrww0k7215jgtw3yfoybv
>>>>> [3]:
>>>>> 
>>>> 
>>> 
>> https://cwiki.apache.org/confluence/display/FLINK/Flink+Release+Management
>>>>> [4]: https://cwiki.apache.org/confluence/display/FLINK/1.18+Release
>>>>> 
>>>> 
>>> 
>> 

Re: [DISCUSS] Update Flink Roadmap

[Jark Wu <im...@gmail.com>]

Hi all,

Sorry for taking so long back here.

Martijn and I have drafted the first version of the updated roadmap,
including the updated feature radar reflecting the current state of
different components.
https://docs.google.com/document/d/12BDiVKEsY-f7HI3suO_IxwzCmR04QcVqLarXgyJAb7c/edit

Feel free to leave comments in the thread or the document.
We may miss mentioning something important, so your help in enriching
the content is greatly appreciated.

Best,
Jark & Martijn


On Fri, 2 Jun 2023 at 00:50, Jing Ge <ji...@ververica.com.invalid> wrote:

> Hi Jark,
>
> Fair enough. Let's do it like you suggested. Thanks!
>
> Best regards,
> Jing
>
> On Thu, Jun 1, 2023 at 6:00 PM Jark Wu <im...@gmail.com> wrote:
>
> > Hi Jing,
> >
> > This thread is for discussing the roadmap for versions 1.18, 2.0, and
> even
> > more.
> > One of the outcomes of this discussion will be an updated version of the
> > current roadmap.
> > Let's work together on refining the roadmap in this thread.
> >
> > Best,
> > Jark
> >
> > On Thu, 1 Jun 2023 at 23:25, Jing Ge <ji...@ververica.com.invalid> wrote:
> >
> > > Hi Jark,
> > >
> > > Thanks for driving it! For point 2, since we are developing 1.18 now,
> > > does it make sense to update the roadmap this time while we are
> releasing
> > > 1.18? This discussion thread will be focusing on the Flink 2.0 roadmap,
> > as
> > > you mentioned previously. WDYT?
> > >
> > > Best regards,
> > > Jing
> > >
> > > On Thu, Jun 1, 2023 at 3:31 PM Jark Wu <im...@gmail.com> wrote:
> > >
> > > > Hi all,
> > > >
> > > > Martijn and I would like to initiate a discussion on the Flink
> roadmap,
> > > > which should cover the project's long-term roadmap and the regular
> > update
> > > > mechanism.
> > > >
> > > > Xintong has already started a discussion about Flink 2.0 planning.
> One
> > of
> > > > the points raised in that discussion is that we should have a
> > high-level
> > > > discussion of the roadmap to present where the project is heading
> > (which
> > > > doesn't necessarily need to block the Flink 2.0 planning). Moreover,
> > the
> > > > roadmap on the Flink website [1] hasn't been updated for half a year,
> > and
> > > > the last update was for the feature radar for the 1.15 release. It
> has
> > > been
> > > > 2 years since the community discussed Flink's overall roadmap.
> > > >
> > > > I would like to raise two topics for discussion:
> > > >
> > > > 1. The new roadmap. This should be an updated version of the current
> > > > roadmap[1].
> > > > 2. A mechanism to regularly discuss and update the roadmap.
> > > >
> > > > To make the first topic discussion more efficient, Martijn and I
> > > volunteer
> > > > to summarize the ongoing big things of different components and
> > present a
> > > > roadmap draft to the community in the next few weeks. This should be
> a
> > > good
> > > > starting point for a more detailed discussion.
> > > >
> > > > Regarding the regular update mechanism, there was a proposal in a
> > thread
> > > > [2] three years ago to make the release manager responsible for
> > updating
> > > > the roadmap. However, it appears that this was not documented as a
> > > release
> > > > management task [3], and the roadmap update wasn't performed for
> > releases
> > > > 1.16 and 1.17.
> > > >
> > > > In my opinion, making release managers responsible for keeping the
> > > roadmap
> > > > up to date is a good idea. Specifically, release managers of release
> X
> > > can
> > > > kick off the roadmap update at the beginning of release X, which can
> > be a
> > > > joint task with collecting a feature list [4]. Additionally, release
> > > > managers of release X-1 can help verify and remove the accomplished
> > items
> > > > from the roadmap and update the feature radar.
> > > >
> > > > What do you think? Do you have other ideas?
> > > >
> > > > Best,
> > > > Jark & Martijn
> > > >
> > > > [1]: https://flink.apache.org/roadmap.html
> > > > [2]:
> https://lists.apache.org/thread/o0l3cg6yphxwrww0k7215jgtw3yfoybv
> > > > [3]:
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/Flink+Release+Management
> > > > [4]: https://cwiki.apache.org/confluence/display/FLINK/1.18+Release
> > > >
> > >
> >
>

Re: [DISCUSS] Update Flink Roadmap

[Jing Ge <ji...@ververica.com.INVALID>]

Hi Jark,

Fair enough. Let's do it like you suggested. Thanks!

Best regards,
Jing

On Thu, Jun 1, 2023 at 6:00 PM Jark Wu <im...@gmail.com> wrote:

> Hi Jing,
>
> This thread is for discussing the roadmap for versions 1.18, 2.0, and even
> more.
> One of the outcomes of this discussion will be an updated version of the
> current roadmap.
> Let's work together on refining the roadmap in this thread.
>
> Best,
> Jark
>
> On Thu, 1 Jun 2023 at 23:25, Jing Ge <ji...@ververica.com.invalid> wrote:
>
> > Hi Jark,
> >
> > Thanks for driving it! For point 2, since we are developing 1.18 now,
> > does it make sense to update the roadmap this time while we are releasing
> > 1.18? This discussion thread will be focusing on the Flink 2.0 roadmap,
> as
> > you mentioned previously. WDYT?
> >
> > Best regards,
> > Jing
> >
> > On Thu, Jun 1, 2023 at 3:31 PM Jark Wu <im...@gmail.com> wrote:
> >
> > > Hi all,
> > >
> > > Martijn and I would like to initiate a discussion on the Flink roadmap,
> > > which should cover the project's long-term roadmap and the regular
> update
> > > mechanism.
> > >
> > > Xintong has already started a discussion about Flink 2.0 planning. One
> of
> > > the points raised in that discussion is that we should have a
> high-level
> > > discussion of the roadmap to present where the project is heading
> (which
> > > doesn't necessarily need to block the Flink 2.0 planning). Moreover,
> the
> > > roadmap on the Flink website [1] hasn't been updated for half a year,
> and
> > > the last update was for the feature radar for the 1.15 release. It has
> > been
> > > 2 years since the community discussed Flink's overall roadmap.
> > >
> > > I would like to raise two topics for discussion:
> > >
> > > 1. The new roadmap. This should be an updated version of the current
> > > roadmap[1].
> > > 2. A mechanism to regularly discuss and update the roadmap.
> > >
> > > To make the first topic discussion more efficient, Martijn and I
> > volunteer
> > > to summarize the ongoing big things of different components and
> present a
> > > roadmap draft to the community in the next few weeks. This should be a
> > good
> > > starting point for a more detailed discussion.
> > >
> > > Regarding the regular update mechanism, there was a proposal in a
> thread
> > > [2] three years ago to make the release manager responsible for
> updating
> > > the roadmap. However, it appears that this was not documented as a
> > release
> > > management task [3], and the roadmap update wasn't performed for
> releases
> > > 1.16 and 1.17.
> > >
> > > In my opinion, making release managers responsible for keeping the
> > roadmap
> > > up to date is a good idea. Specifically, release managers of release X
> > can
> > > kick off the roadmap update at the beginning of release X, which can
> be a
> > > joint task with collecting a feature list [4]. Additionally, release
> > > managers of release X-1 can help verify and remove the accomplished
> items
> > > from the roadmap and update the feature radar.
> > >
> > > What do you think? Do you have other ideas?
> > >
> > > Best,
> > > Jark & Martijn
> > >
> > > [1]: https://flink.apache.org/roadmap.html
> > > [2]: https://lists.apache.org/thread/o0l3cg6yphxwrww0k7215jgtw3yfoybv
> > > [3]:
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/Flink+Release+Management
> > > [4]: https://cwiki.apache.org/confluence/display/FLINK/1.18+Release
> > >
> >
>

Re: [DISCUSS] Update Flink Roadmap

[Jark Wu <im...@gmail.com>]

Hi Jing,

This thread is for discussing the roadmap for versions 1.18, 2.0, and even
more.
One of the outcomes of this discussion will be an updated version of the
current roadmap.
Let's work together on refining the roadmap in this thread.

Best,
Jark

On Thu, 1 Jun 2023 at 23:25, Jing Ge <ji...@ververica.com.invalid> wrote:

> Hi Jark,
>
> Thanks for driving it! For point 2, since we are developing 1.18 now,
> does it make sense to update the roadmap this time while we are releasing
> 1.18? This discussion thread will be focusing on the Flink 2.0 roadmap, as
> you mentioned previously. WDYT?
>
> Best regards,
> Jing
>
> On Thu, Jun 1, 2023 at 3:31 PM Jark Wu <im...@gmail.com> wrote:
>
> > Hi all,
> >
> > Martijn and I would like to initiate a discussion on the Flink roadmap,
> > which should cover the project's long-term roadmap and the regular update
> > mechanism.
> >
> > Xintong has already started a discussion about Flink 2.0 planning. One of
> > the points raised in that discussion is that we should have a high-level
> > discussion of the roadmap to present where the project is heading (which
> > doesn't necessarily need to block the Flink 2.0 planning). Moreover, the
> > roadmap on the Flink website [1] hasn't been updated for half a year, and
> > the last update was for the feature radar for the 1.15 release. It has
> been
> > 2 years since the community discussed Flink's overall roadmap.
> >
> > I would like to raise two topics for discussion:
> >
> > 1. The new roadmap. This should be an updated version of the current
> > roadmap[1].
> > 2. A mechanism to regularly discuss and update the roadmap.
> >
> > To make the first topic discussion more efficient, Martijn and I
> volunteer
> > to summarize the ongoing big things of different components and present a
> > roadmap draft to the community in the next few weeks. This should be a
> good
> > starting point for a more detailed discussion.
> >
> > Regarding the regular update mechanism, there was a proposal in a thread
> > [2] three years ago to make the release manager responsible for updating
> > the roadmap. However, it appears that this was not documented as a
> release
> > management task [3], and the roadmap update wasn't performed for releases
> > 1.16 and 1.17.
> >
> > In my opinion, making release managers responsible for keeping the
> roadmap
> > up to date is a good idea. Specifically, release managers of release X
> can
> > kick off the roadmap update at the beginning of release X, which can be a
> > joint task with collecting a feature list [4]. Additionally, release
> > managers of release X-1 can help verify and remove the accomplished items
> > from the roadmap and update the feature radar.
> >
> > What do you think? Do you have other ideas?
> >
> > Best,
> > Jark & Martijn
> >
> > [1]: https://flink.apache.org/roadmap.html
> > [2]: https://lists.apache.org/thread/o0l3cg6yphxwrww0k7215jgtw3yfoybv
> > [3]:
> >
> https://cwiki.apache.org/confluence/display/FLINK/Flink+Release+Management
> > [4]: https://cwiki.apache.org/confluence/display/FLINK/1.18+Release
> >
>

Re: [DISCUSS] Update Flink Roadmap

[Jing Ge <ji...@ververica.com.INVALID>]

Hi Jark,

Thanks for driving it! For point 2, since we are developing 1.18 now,
does it make sense to update the roadmap this time while we are releasing
1.18? This discussion thread will be focusing on the Flink 2.0 roadmap, as
you mentioned previously. WDYT?

Best regards,
Jing

On Thu, Jun 1, 2023 at 3:31 PM Jark Wu <im...@gmail.com> wrote:

> Hi all,
>
> Martijn and I would like to initiate a discussion on the Flink roadmap,
> which should cover the project's long-term roadmap and the regular update
> mechanism.
>
> Xintong has already started a discussion about Flink 2.0 planning. One of
> the points raised in that discussion is that we should have a high-level
> discussion of the roadmap to present where the project is heading (which
> doesn't necessarily need to block the Flink 2.0 planning). Moreover, the
> roadmap on the Flink website [1] hasn't been updated for half a year, and
> the last update was for the feature radar for the 1.15 release. It has been
> 2 years since the community discussed Flink's overall roadmap.
>
> I would like to raise two topics for discussion:
>
> 1. The new roadmap. This should be an updated version of the current
> roadmap[1].
> 2. A mechanism to regularly discuss and update the roadmap.
>
> To make the first topic discussion more efficient, Martijn and I volunteer
> to summarize the ongoing big things of different components and present a
> roadmap draft to the community in the next few weeks. This should be a good
> starting point for a more detailed discussion.
>
> Regarding the regular update mechanism, there was a proposal in a thread
> [2] three years ago to make the release manager responsible for updating
> the roadmap. However, it appears that this was not documented as a release
> management task [3], and the roadmap update wasn't performed for releases
> 1.16 and 1.17.
>
> In my opinion, making release managers responsible for keeping the roadmap
> up to date is a good idea. Specifically, release managers of release X can
> kick off the roadmap update at the beginning of release X, which can be a
> joint task with collecting a feature list [4]. Additionally, release
> managers of release X-1 can help verify and remove the accomplished items
> from the roadmap and update the feature radar.
>
> What do you think? Do you have other ideas?
>
> Best,
> Jark & Martijn
>
> [1]: https://flink.apache.org/roadmap.html
> [2]: https://lists.apache.org/thread/o0l3cg6yphxwrww0k7215jgtw3yfoybv
> [3]:
> https://cwiki.apache.org/confluence/display/FLINK/Flink+Release+Management
> [4]: https://cwiki.apache.org/confluence/display/FLINK/1.18+Release
>