[PROPOSAL] Contribute Stateful Functions to Apache Flink

[Stephan Ewen <se...@apache.org>]

Dear Flink Community!

Some of you probably heard it already: On Tuesday, at Flink Forward Berlin,
we announced **Stateful Functions**.

Stateful Functions is a library on Flink to implement general purpose
applications. It is built around stateful functions (who would have thunk)
that can communicate arbitrarily through messages, have consistent state,
and a small resource footprint. They are a bit like keyed ProcessFunctions
that can send each other messages.
As simple as this sounds, this means you can now communicate in non-DAG
patterns, so it allows users to build programs they cannot build with Flink.
It also has other neat properties, like multiplexing of functions, modular
composition, tooling both container-based deployments and as-a-Flink-job
deployments.

You can find out more about it here
  - Website: https://statefun.io/
  - Code: https://github.com/ververica/stateful-functions
  - Talk with motivation:
https://speakerdeck.com/stephanewen/stateful-functions-building-general-purpose-applications-and-services-on-apache-flink?slide=12


Now for the main issue: **We would like to contribute this project to
Apache Flink**

I believe that this is a great fit for both sides.
For the Flink community, it would be a way to extend the capabilities and
use cases of Flink into a completely different type of applications and
thus grow the community into this new field.
Many discussions recently about evolving the Flink runtime (both on the
mailing list and at conferences) show the interest in Flink users in the
space that Stateful Functions covers.
It seems natural that Stateful Functions should closely co-develop with
Apache Flink, ideally as part of the project.

There are many details to be discusses, for example whether this should be
added to the Flink core repository, or whether we and to create a separate
repository
for this. But I think we should start discussing this after we have
consensus on whether the community wants this contribution.

Really looking forward to hear what you think!

Best Regards,
Stephan

Re: [PROPOSAL] Contribute Stateful Functions to Apache Flink

[Jark Wu <im...@gmail.com>]

Hi Stephan,

big +1 for adding statefun to Apache Flink.

I think we can add it in the main repository as a library just like State
Process API.
Having it in the main repository can closely co-develop with Apache Flink
which be beneficial for both side.

Regards,
Jark


On Fri, 11 Oct 2019 at 19:12, Till Rohrmann <tr...@apache.org> wrote:

> Hi Stephan,
>
> +1 for adding stateful functions to Flink. I believe the new set of
> applications this feature will unlock will be super interesting for new and
> existing Flink users alike.
>
> One reason for not including it in the main repository would to not being
> bound to Flink's release cadence. This would allow to release faster and
> more often. However, I believe that having it eventually in Flink's main
> repository would be beneficial in the long run.
>
> Cheers,
> Till
>
> On Fri, Oct 11, 2019 at 12:56 PM Trevor Grant <tr...@gmail.com>
> wrote:
>
>> +1 non-binding on contribution.
>>
>> Separate repo, or feature branch to start maybe? I just feel like in the
>> beginning this thing is going to have lots of breaking changes that maybe
>> aren't going to fit well with tests / other "v1+" release code. Just my
>> .02.
>>
>>
>>
>> On Fri, Oct 11, 2019 at 4:38 AM Stephan Ewen <se...@apache.org> wrote:
>>
>>> Dear Flink Community!
>>>
>>> Some of you probably heard it already: On Tuesday, at Flink Forward
>>> Berlin, we announced **Stateful Functions**.
>>>
>>> Stateful Functions is a library on Flink to implement general purpose
>>> applications. It is built around stateful functions (who would have thunk)
>>> that can communicate arbitrarily through messages, have consistent
>>> state, and a small resource footprint. They are a bit like keyed
>>> ProcessFunctions
>>> that can send each other messages.
>>> As simple as this sounds, this means you can now communicate in non-DAG
>>> patterns, so it allows users to build programs they cannot build with Flink.
>>> It also has other neat properties, like multiplexing of functions,
>>> modular composition, tooling both container-based deployments and
>>> as-a-Flink-job deployments.
>>>
>>> You can find out more about it here
>>>   - Website: https://statefun.io/
>>>   - Code: https://github.com/ververica/stateful-functions
>>>   - Talk with motivation:
>>> https://speakerdeck.com/stephanewen/stateful-functions-building-general-purpose-applications-and-services-on-apache-flink?slide=12
>>>
>>>
>>> Now for the main issue: **We would like to contribute this project to
>>> Apache Flink**
>>>
>>> I believe that this is a great fit for both sides.
>>> For the Flink community, it would be a way to extend the capabilities
>>> and use cases of Flink into a completely different type of applications and
>>> thus grow the community into this new field.
>>> Many discussions recently about evolving the Flink runtime (both on the
>>> mailing list and at conferences) show the interest in Flink users in the
>>> space that Stateful Functions covers.
>>> It seems natural that Stateful Functions should closely co-develop with
>>> Apache Flink, ideally as part of the project.
>>>
>>> There are many details to be discusses, for example whether this should
>>> be added to the Flink core repository, or whether we and to create a
>>> separate repository
>>> for this. But I think we should start discussing this after we have
>>> consensus on whether the community wants this contribution.
>>>
>>> Really looking forward to hear what you think!
>>>
>>> Best Regards,
>>> Stephan
>>>
>>>

Re: [PROPOSAL] Contribute Stateful Functions to Apache Flink

[Jark Wu <im...@gmail.com>]

Hi Stephan,

big +1 for adding statefun to Apache Flink.

I think we can add it in the main repository as a library just like State
Process API.
Having it in the main repository can closely co-develop with Apache Flink
which be beneficial for both side.

Regards,
Jark


On Fri, 11 Oct 2019 at 19:12, Till Rohrmann <tr...@apache.org> wrote:

> Hi Stephan,
>
> +1 for adding stateful functions to Flink. I believe the new set of
> applications this feature will unlock will be super interesting for new and
> existing Flink users alike.
>
> One reason for not including it in the main repository would to not being
> bound to Flink's release cadence. This would allow to release faster and
> more often. However, I believe that having it eventually in Flink's main
> repository would be beneficial in the long run.
>
> Cheers,
> Till
>
> On Fri, Oct 11, 2019 at 12:56 PM Trevor Grant <tr...@gmail.com>
> wrote:
>
>> +1 non-binding on contribution.
>>
>> Separate repo, or feature branch to start maybe? I just feel like in the
>> beginning this thing is going to have lots of breaking changes that maybe
>> aren't going to fit well with tests / other "v1+" release code. Just my
>> .02.
>>
>>
>>
>> On Fri, Oct 11, 2019 at 4:38 AM Stephan Ewen <se...@apache.org> wrote:
>>
>>> Dear Flink Community!
>>>
>>> Some of you probably heard it already: On Tuesday, at Flink Forward
>>> Berlin, we announced **Stateful Functions**.
>>>
>>> Stateful Functions is a library on Flink to implement general purpose
>>> applications. It is built around stateful functions (who would have thunk)
>>> that can communicate arbitrarily through messages, have consistent
>>> state, and a small resource footprint. They are a bit like keyed
>>> ProcessFunctions
>>> that can send each other messages.
>>> As simple as this sounds, this means you can now communicate in non-DAG
>>> patterns, so it allows users to build programs they cannot build with Flink.
>>> It also has other neat properties, like multiplexing of functions,
>>> modular composition, tooling both container-based deployments and
>>> as-a-Flink-job deployments.
>>>
>>> You can find out more about it here
>>>   - Website: https://statefun.io/
>>>   - Code: https://github.com/ververica/stateful-functions
>>>   - Talk with motivation:
>>> https://speakerdeck.com/stephanewen/stateful-functions-building-general-purpose-applications-and-services-on-apache-flink?slide=12
>>>
>>>
>>> Now for the main issue: **We would like to contribute this project to
>>> Apache Flink**
>>>
>>> I believe that this is a great fit for both sides.
>>> For the Flink community, it would be a way to extend the capabilities
>>> and use cases of Flink into a completely different type of applications and
>>> thus grow the community into this new field.
>>> Many discussions recently about evolving the Flink runtime (both on the
>>> mailing list and at conferences) show the interest in Flink users in the
>>> space that Stateful Functions covers.
>>> It seems natural that Stateful Functions should closely co-develop with
>>> Apache Flink, ideally as part of the project.
>>>
>>> There are many details to be discusses, for example whether this should
>>> be added to the Flink core repository, or whether we and to create a
>>> separate repository
>>> for this. But I think we should start discussing this after we have
>>> consensus on whether the community wants this contribution.
>>>
>>> Really looking forward to hear what you think!
>>>
>>> Best Regards,
>>> Stephan
>>>
>>>

Re: [PROPOSAL] Contribute Stateful Functions to Apache Flink

[Kostas Kloudas <kk...@gmail.com>]

Hi all,

Big +1 for contributing Stateful Functions to Flink and as for the
main question at hand, I would vote for putting it in the main
repository.

I understand that this can couple the release cadence of Flink and
Stateful Functions although I think the pros of having a "you break
it,
you fix it" policy outperform the cons of tying the release cadences.

Looking forward to the integration and the new usecases it may bring!

Cheers,
Kostas

On Mon, Oct 14, 2019 at 9:35 AM Dian Fu <di...@gmail.com> wrote:
>
> Hi Stephan,
>
> Big +1 for adding stateful functions to Apache Flink! The use cases unlocked with this feature are very interesting and promising.
>
> Regarding to whether to place it into Flink core repository, personally I perfer to put it in the main repository. This feature introduces a new set of APIs and it will support a new set of applications. It enriches the API stack of Apache Flink. This is somewhat simlar to the Table API & SQL, State Processor API, CEP library, etc. If the applications supported by this feature are important enough for Flink, it's more appropriate to put it directly into the main repository.
>
> Regards,
> Dian
>
> > 在 2019年10月13日，上午10:47，Hequn Cheng <ch...@gmail.com> 写道：
> >
> > Hi Stephan,
> >
> > Big +1 for adding this to Apache Flink!
> >
> > As for the problem of whether this should be added to the Flink main repository, from my side, I prefer to put it in the main repository. Not only Stateful Functions shares very close relations with the current Flink, but also other libs or modules in Flink can make use of it the other way round in the future. At that time the Flink API stack would also be changed a bit and this would be cool.
> >
> > Best, Hequn
> >
> > On Sat, Oct 12, 2019 at 9:16 PM Biao Liu <mmyy1110@gmail.com <ma...@gmail.com>> wrote:
> > Hi Stehpan,
> >
> > +1 for having Stateful Functions in Flink.
> >
> > Before discussing which repository it should belong, I was wondering if we have reached an agreement of "splitting flink repository" as Piotr mentioned or not. It seems that it's just no more further discussion.
> > It's OK for me to add it to core repository. After all almost everything is in core repository now. But if we decide to split the core repository someday, I tend to create a separate repository for Stateful Functions. It might be good time to take the first step of splitting.
> >
> > Thanks,
> > Biao /'bɪ.aʊ/
> >
> >
> >
> > On Sat, 12 Oct 2019 at 19:31, Yu Li <carp84@gmail.com <ma...@gmail.com>> wrote:
> > Hi Stephan,
> >
> > Big +1 for adding stateful functions to Flink. I believe a lot of user would be interested to try this out and I could imagine how this could contribute to reduce the TCO for business requiring both streaming processing and stateful functions.
> >
> > And my 2 cents is to put it into flink core repository since I could see a tight connection between this library and flink state.
> >
> > Best Regards,
> > Yu
> >
> >
> > On Sat, 12 Oct 2019 at 17:31, jincheng sun <sunjincheng121@gmail.com <ma...@gmail.com>> wrote:
> > Hi Stephan,
> >
> > bit +1 for adding this great features to Apache Flink.
> >
> > Regarding where we should place it, put it into Flink core repository or create a separate repository? I prefer put it into main repository and looking forward the more detail discussion for this decision.
> >
> > Best,
> > Jincheng
> >
> >
> > Jingsong Li <jingsonglee0@gmail.com <ma...@gmail.com>> 于2019年10月12日周六 上午11:32写道：
> > Hi Stephan,
> >
> > big +1 for this contribution. It provides another user interface that is easy to use and popular at this time. these functions, It's hard for users to write in SQL/TableApi, while using DataStream is too complex. (We've done some stateFun kind jobs using DataStream before). With statefun, it is very easy.
> >
> > I think it's also a good opportunity to exercise Flink's core capabilities. I looked at stateful-functions-flink briefly, it is very interesting. I think there are many other things Flink can improve. So I think it's a better thing to put it into Flink, and the improvement for it will be more natural in the future.
> >
> > Best,
> > Jingsong Lee
> >
> > On Fri, Oct 11, 2019 at 7:33 PM Dawid Wysakowicz <dwysakowicz@apache.org <ma...@apache.org>> wrote:
> > Hi Stephan,
> >
> > I think this is a nice library, but what I like more about it is that it suggests exploring different use-cases. I think it definitely makes sense for the Flink community to explore more lightweight applications that reuses resources. Therefore I definitely think it is a good idea for Flink community to accept this contribution and help maintaining it.
> >
> > Personally I'd prefer to have it in a separate repository. There were a few discussions before where different people were suggesting to extract connectors and other libraries to separate repositories. Moreover I think it could serve as an example for the Flink ecosystem website[1]. This could be the first project in there and give a good impression that the community sees potential in the ecosystem website.
> > Lastly, I'm wondering if this should go through PMC vote according to our bylaws[2]. In the end the suggestion is to adopt an existing code base as is. It also proposes a new programs concept that could result in a shift of priorities for the community in a long run.
> > Best,
> >
> > Dawid
> > [1] http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Create-a-Flink-ecosystem-website-td27519.html <http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Create-a-Flink-ecosystem-website-td27519.html>
> > [2] https://cwiki.apache.org/confluence/display/FLINK/Flink+Bylaws <https://cwiki.apache.org/confluence/display/FLINK/Flink+Bylaws>
> > On 11/10/2019 13:12, Till Rohrmann wrote:
> >> Hi Stephan,
> >>
> >> +1 for adding stateful functions to Flink. I believe the new set of applications this feature will unlock will be super interesting for new and existing Flink users alike.
> >>
> >> One reason for not including it in the main repository would to not being bound to Flink's release cadence. This would allow to release faster and more often. However, I believe that having it eventually in Flink's main repository would be beneficial in the long run.
> >>
> >> Cheers,
> >> Till
> >>
> >> On Fri, Oct 11, 2019 at 12:56 PM Trevor Grant <trevor.d.grant@gmail.com <ma...@gmail.com>> wrote:
> >> +1 non-binding on contribution.
> >>
> >> Separate repo, or feature branch to start maybe? I just feel like in the beginning this thing is going to have lots of breaking changes that maybe aren't going to fit well with tests / other "v1+" release code. Just my .02.
> >>
> >>
> >>
> >> On Fri, Oct 11, 2019 at 4:38 AM Stephan Ewen <sewen@apache.org <ma...@apache.org>> wrote:
> >> Dear Flink Community!
> >>
> >> Some of you probably heard it already: On Tuesday, at Flink Forward Berlin, we announced **Stateful Functions**.
> >>
> >> Stateful Functions is a library on Flink to implement general purpose applications. It is built around stateful functions (who would have thunk)
> >> that can communicate arbitrarily through messages, have consistent state, and a small resource footprint. They are a bit like keyed ProcessFunctions
> >> that can send each other messages.
> >> As simple as this sounds, this means you can now communicate in non-DAG patterns, so it allows users to build programs they cannot build with Flink.
> >> It also has other neat properties, like multiplexing of functions, modular composition, tooling both container-based deployments and as-a-Flink-job deployments.
> >>
> >> You can find out more about it here
> >>   - Website: https://statefun.io/ <https://statefun.io/>
> >>   - Code: https://github.com/ververica/stateful-functions <https://github.com/ververica/stateful-functions>
> >>   - Talk with motivation: https://speakerdeck.com/stephanewen/stateful-functions-building-general-purpose-applications-and-services-on-apache-flink?slide=12 <https://speakerdeck.com/stephanewen/stateful-functions-building-general-purpose-applications-and-services-on-apache-flink?slide=12>
> >>
> >>
> >> Now for the main issue: **We would like to contribute this project to Apache Flink**
> >>
> >> I believe that this is a great fit for both sides.
> >> For the Flink community, it would be a way to extend the capabilities and use cases of Flink into a completely different type of applications and thus grow the community into this new field.
> >> Many discussions recently about evolving the Flink runtime (both on the mailing list and at conferences) show the interest in Flink users in the space that Stateful Functions covers.
> >> It seems natural that Stateful Functions should closely co-develop with Apache Flink, ideally as part of the project.
> >>
> >> There are many details to be discusses, for example whether this should be added to the Flink core repository, or whether we and to create a separate repository
> >> for this. But I think we should start discussing this after we have consensus on whether the community wants this contribution.
> >>
> >> Really looking forward to hear what you think!
> >>
> >> Best Regards,
> >> Stephan
> >>
> >
> >
> > --
> > Best, Jingsong Lee
>

Re: [PROPOSAL] Contribute Stateful Functions to Apache Flink

[Kostas Kloudas <kk...@gmail.com>]

Hi all,

Big +1 for contributing Stateful Functions to Flink and as for the
main question at hand, I would vote for putting it in the main
repository.

I understand that this can couple the release cadence of Flink and
Stateful Functions although I think the pros of having a "you break
it,
you fix it" policy outperform the cons of tying the release cadences.

Looking forward to the integration and the new usecases it may bring!

Cheers,
Kostas

On Mon, Oct 14, 2019 at 9:35 AM Dian Fu <di...@gmail.com> wrote:
>
> Hi Stephan,
>
> Big +1 for adding stateful functions to Apache Flink! The use cases unlocked with this feature are very interesting and promising.
>
> Regarding to whether to place it into Flink core repository, personally I perfer to put it in the main repository. This feature introduces a new set of APIs and it will support a new set of applications. It enriches the API stack of Apache Flink. This is somewhat simlar to the Table API & SQL, State Processor API, CEP library, etc. If the applications supported by this feature are important enough for Flink, it's more appropriate to put it directly into the main repository.
>
> Regards,
> Dian
>
> > 在 2019年10月13日，上午10:47，Hequn Cheng <ch...@gmail.com> 写道：
> >
> > Hi Stephan,
> >
> > Big +1 for adding this to Apache Flink!
> >
> > As for the problem of whether this should be added to the Flink main repository, from my side, I prefer to put it in the main repository. Not only Stateful Functions shares very close relations with the current Flink, but also other libs or modules in Flink can make use of it the other way round in the future. At that time the Flink API stack would also be changed a bit and this would be cool.
> >
> > Best, Hequn
> >
> > On Sat, Oct 12, 2019 at 9:16 PM Biao Liu <mmyy1110@gmail.com <ma...@gmail.com>> wrote:
> > Hi Stehpan,
> >
> > +1 for having Stateful Functions in Flink.
> >
> > Before discussing which repository it should belong, I was wondering if we have reached an agreement of "splitting flink repository" as Piotr mentioned or not. It seems that it's just no more further discussion.
> > It's OK for me to add it to core repository. After all almost everything is in core repository now. But if we decide to split the core repository someday, I tend to create a separate repository for Stateful Functions. It might be good time to take the first step of splitting.
> >
> > Thanks,
> > Biao /'bɪ.aʊ/
> >
> >
> >
> > On Sat, 12 Oct 2019 at 19:31, Yu Li <carp84@gmail.com <ma...@gmail.com>> wrote:
> > Hi Stephan,
> >
> > Big +1 for adding stateful functions to Flink. I believe a lot of user would be interested to try this out and I could imagine how this could contribute to reduce the TCO for business requiring both streaming processing and stateful functions.
> >
> > And my 2 cents is to put it into flink core repository since I could see a tight connection between this library and flink state.
> >
> > Best Regards,
> > Yu
> >
> >
> > On Sat, 12 Oct 2019 at 17:31, jincheng sun <sunjincheng121@gmail.com <ma...@gmail.com>> wrote:
> > Hi Stephan,
> >
> > bit +1 for adding this great features to Apache Flink.
> >
> > Regarding where we should place it, put it into Flink core repository or create a separate repository? I prefer put it into main repository and looking forward the more detail discussion for this decision.
> >
> > Best,
> > Jincheng
> >
> >
> > Jingsong Li <jingsonglee0@gmail.com <ma...@gmail.com>> 于2019年10月12日周六 上午11:32写道：
> > Hi Stephan,
> >
> > big +1 for this contribution. It provides another user interface that is easy to use and popular at this time. these functions, It's hard for users to write in SQL/TableApi, while using DataStream is too complex. (We've done some stateFun kind jobs using DataStream before). With statefun, it is very easy.
> >
> > I think it's also a good opportunity to exercise Flink's core capabilities. I looked at stateful-functions-flink briefly, it is very interesting. I think there are many other things Flink can improve. So I think it's a better thing to put it into Flink, and the improvement for it will be more natural in the future.
> >
> > Best,
> > Jingsong Lee
> >
> > On Fri, Oct 11, 2019 at 7:33 PM Dawid Wysakowicz <dwysakowicz@apache.org <ma...@apache.org>> wrote:
> > Hi Stephan,
> >
> > I think this is a nice library, but what I like more about it is that it suggests exploring different use-cases. I think it definitely makes sense for the Flink community to explore more lightweight applications that reuses resources. Therefore I definitely think it is a good idea for Flink community to accept this contribution and help maintaining it.
> >
> > Personally I'd prefer to have it in a separate repository. There were a few discussions before where different people were suggesting to extract connectors and other libraries to separate repositories. Moreover I think it could serve as an example for the Flink ecosystem website[1]. This could be the first project in there and give a good impression that the community sees potential in the ecosystem website.
> > Lastly, I'm wondering if this should go through PMC vote according to our bylaws[2]. In the end the suggestion is to adopt an existing code base as is. It also proposes a new programs concept that could result in a shift of priorities for the community in a long run.
> > Best,
> >
> > Dawid
> > [1] http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Create-a-Flink-ecosystem-website-td27519.html <http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Create-a-Flink-ecosystem-website-td27519.html>
> > [2] https://cwiki.apache.org/confluence/display/FLINK/Flink+Bylaws <https://cwiki.apache.org/confluence/display/FLINK/Flink+Bylaws>
> > On 11/10/2019 13:12, Till Rohrmann wrote:
> >> Hi Stephan,
> >>
> >> +1 for adding stateful functions to Flink. I believe the new set of applications this feature will unlock will be super interesting for new and existing Flink users alike.
> >>
> >> One reason for not including it in the main repository would to not being bound to Flink's release cadence. This would allow to release faster and more often. However, I believe that having it eventually in Flink's main repository would be beneficial in the long run.
> >>
> >> Cheers,
> >> Till
> >>
> >> On Fri, Oct 11, 2019 at 12:56 PM Trevor Grant <trevor.d.grant@gmail.com <ma...@gmail.com>> wrote:
> >> +1 non-binding on contribution.
> >>
> >> Separate repo, or feature branch to start maybe? I just feel like in the beginning this thing is going to have lots of breaking changes that maybe aren't going to fit well with tests / other "v1+" release code. Just my .02.
> >>
> >>
> >>
> >> On Fri, Oct 11, 2019 at 4:38 AM Stephan Ewen <sewen@apache.org <ma...@apache.org>> wrote:
> >> Dear Flink Community!
> >>
> >> Some of you probably heard it already: On Tuesday, at Flink Forward Berlin, we announced **Stateful Functions**.
> >>
> >> Stateful Functions is a library on Flink to implement general purpose applications. It is built around stateful functions (who would have thunk)
> >> that can communicate arbitrarily through messages, have consistent state, and a small resource footprint. They are a bit like keyed ProcessFunctions
> >> that can send each other messages.
> >> As simple as this sounds, this means you can now communicate in non-DAG patterns, so it allows users to build programs they cannot build with Flink.
> >> It also has other neat properties, like multiplexing of functions, modular composition, tooling both container-based deployments and as-a-Flink-job deployments.
> >>
> >> You can find out more about it here
> >>   - Website: https://statefun.io/ <https://statefun.io/>
> >>   - Code: https://github.com/ververica/stateful-functions <https://github.com/ververica/stateful-functions>
> >>   - Talk with motivation: https://speakerdeck.com/stephanewen/stateful-functions-building-general-purpose-applications-and-services-on-apache-flink?slide=12 <https://speakerdeck.com/stephanewen/stateful-functions-building-general-purpose-applications-and-services-on-apache-flink?slide=12>
> >>
> >>
> >> Now for the main issue: **We would like to contribute this project to Apache Flink**
> >>
> >> I believe that this is a great fit for both sides.
> >> For the Flink community, it would be a way to extend the capabilities and use cases of Flink into a completely different type of applications and thus grow the community into this new field.
> >> Many discussions recently about evolving the Flink runtime (both on the mailing list and at conferences) show the interest in Flink users in the space that Stateful Functions covers.
> >> It seems natural that Stateful Functions should closely co-develop with Apache Flink, ideally as part of the project.
> >>
> >> There are many details to be discusses, for example whether this should be added to the Flink core repository, or whether we and to create a separate repository
> >> for this. But I think we should start discussing this after we have consensus on whether the community wants this contribution.
> >>
> >> Really looking forward to hear what you think!
> >>
> >> Best Regards,
> >> Stephan
> >>
> >
> >
> > --
> > Best, Jingsong Lee
>

Re: [PROPOSAL] Contribute Stateful Functions to Apache Flink

[Dian Fu <di...@gmail.com>]

Hi Stephan,

Big +1 for adding stateful functions to Apache Flink! The use cases unlocked with this feature are very interesting and promising.

Regarding to whether to place it into Flink core repository, personally I perfer to put it in the main repository. This feature introduces a new set of APIs and it will support a new set of applications. It enriches the API stack of Apache Flink. This is somewhat simlar to the Table API & SQL, State Processor API, CEP library, etc. If the applications supported by this feature are important enough for Flink, it's more appropriate to put it directly into the main repository.

Regards,
Dian

> 在 2019年10月13日，上午10:47，Hequn Cheng <ch...@gmail.com> 写道：
> 
> Hi Stephan,
> 
> Big +1 for adding this to Apache Flink! 
> 
> As for the problem of whether this should be added to the Flink main repository, from my side, I prefer to put it in the main repository. Not only Stateful Functions shares very close relations with the current Flink, but also other libs or modules in Flink can make use of it the other way round in the future. At that time the Flink API stack would also be changed a bit and this would be cool.
> 
> Best, Hequn
> 
> On Sat, Oct 12, 2019 at 9:16 PM Biao Liu <mmyy1110@gmail.com <ma...@gmail.com>> wrote:
> Hi Stehpan,
> 
> +1 for having Stateful Functions in Flink.
> 
> Before discussing which repository it should belong, I was wondering if we have reached an agreement of "splitting flink repository" as Piotr mentioned or not. It seems that it's just no more further discussion. 
> It's OK for me to add it to core repository. After all almost everything is in core repository now. But if we decide to split the core repository someday, I tend to create a separate repository for Stateful Functions. It might be good time to take the first step of splitting.
> 
> Thanks,
> Biao /'bɪ.aʊ/
> 
> 
> 
> On Sat, 12 Oct 2019 at 19:31, Yu Li <carp84@gmail.com <ma...@gmail.com>> wrote:
> Hi Stephan,
> 
> Big +1 for adding stateful functions to Flink. I believe a lot of user would be interested to try this out and I could imagine how this could contribute to reduce the TCO for business requiring both streaming processing and stateful functions.
> 
> And my 2 cents is to put it into flink core repository since I could see a tight connection between this library and flink state.
> 
> Best Regards,
> Yu
> 
> 
> On Sat, 12 Oct 2019 at 17:31, jincheng sun <sunjincheng121@gmail.com <ma...@gmail.com>> wrote:
> Hi Stephan,
> 
> bit +1 for adding this great features to Apache Flink.
> 
> Regarding where we should place it, put it into Flink core repository or create a separate repository? I prefer put it into main repository and looking forward the more detail discussion for this decision.
> 
> Best,
> Jincheng
> 
> 
> Jingsong Li <jingsonglee0@gmail.com <ma...@gmail.com>> 于2019年10月12日周六 上午11:32写道：
> Hi Stephan,
> 
> big +1 for this contribution. It provides another user interface that is easy to use and popular at this time. these functions, It's hard for users to write in SQL/TableApi, while using DataStream is too complex. (We've done some stateFun kind jobs using DataStream before). With statefun, it is very easy.
> 
> I think it's also a good opportunity to exercise Flink's core capabilities. I looked at stateful-functions-flink briefly, it is very interesting. I think there are many other things Flink can improve. So I think it's a better thing to put it into Flink, and the improvement for it will be more natural in the future.
> 
> Best,
> Jingsong Lee
> 
> On Fri, Oct 11, 2019 at 7:33 PM Dawid Wysakowicz <dwysakowicz@apache.org <ma...@apache.org>> wrote:
> Hi Stephan,
> 
> I think this is a nice library, but what I like more about it is that it suggests exploring different use-cases. I think it definitely makes sense for the Flink community to explore more lightweight applications that reuses resources. Therefore I definitely think it is a good idea for Flink community to accept this contribution and help maintaining it.
> 
> Personally I'd prefer to have it in a separate repository. There were a few discussions before where different people were suggesting to extract connectors and other libraries to separate repositories. Moreover I think it could serve as an example for the Flink ecosystem website[1]. This could be the first project in there and give a good impression that the community sees potential in the ecosystem website.
> Lastly, I'm wondering if this should go through PMC vote according to our bylaws[2]. In the end the suggestion is to adopt an existing code base as is. It also proposes a new programs concept that could result in a shift of priorities for the community in a long run.
> Best,
> 
> Dawid
> [1] http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Create-a-Flink-ecosystem-website-td27519.html <http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Create-a-Flink-ecosystem-website-td27519.html>
> [2] https://cwiki.apache.org/confluence/display/FLINK/Flink+Bylaws <https://cwiki.apache.org/confluence/display/FLINK/Flink+Bylaws>
> On 11/10/2019 13:12, Till Rohrmann wrote:
>> Hi Stephan,
>> 
>> +1 for adding stateful functions to Flink. I believe the new set of applications this feature will unlock will be super interesting for new and existing Flink users alike.
>> 
>> One reason for not including it in the main repository would to not being bound to Flink's release cadence. This would allow to release faster and more often. However, I believe that having it eventually in Flink's main repository would be beneficial in the long run.
>> 
>> Cheers,
>> Till
>> 
>> On Fri, Oct 11, 2019 at 12:56 PM Trevor Grant <trevor.d.grant@gmail.com <ma...@gmail.com>> wrote:
>> +1 non-binding on contribution. 
>> 
>> Separate repo, or feature branch to start maybe? I just feel like in the beginning this thing is going to have lots of breaking changes that maybe aren't going to fit well with tests / other "v1+" release code. Just my .02. 
>> 
>> 
>> 
>> On Fri, Oct 11, 2019 at 4:38 AM Stephan Ewen <sewen@apache.org <ma...@apache.org>> wrote:
>> Dear Flink Community!
>> 
>> Some of you probably heard it already: On Tuesday, at Flink Forward Berlin, we announced **Stateful Functions**.
>> 
>> Stateful Functions is a library on Flink to implement general purpose applications. It is built around stateful functions (who would have thunk)
>> that can communicate arbitrarily through messages, have consistent state, and a small resource footprint. They are a bit like keyed ProcessFunctions
>> that can send each other messages.
>> As simple as this sounds, this means you can now communicate in non-DAG patterns, so it allows users to build programs they cannot build with Flink.
>> It also has other neat properties, like multiplexing of functions, modular composition, tooling both container-based deployments and as-a-Flink-job deployments.
>> 
>> You can find out more about it here
>>   - Website: https://statefun.io/ <https://statefun.io/>
>>   - Code: https://github.com/ververica/stateful-functions <https://github.com/ververica/stateful-functions>
>>   - Talk with motivation: https://speakerdeck.com/stephanewen/stateful-functions-building-general-purpose-applications-and-services-on-apache-flink?slide=12 <https://speakerdeck.com/stephanewen/stateful-functions-building-general-purpose-applications-and-services-on-apache-flink?slide=12>
>> 
>> 
>> Now for the main issue: **We would like to contribute this project to Apache Flink**
>> 
>> I believe that this is a great fit for both sides.
>> For the Flink community, it would be a way to extend the capabilities and use cases of Flink into a completely different type of applications and thus grow the community into this new field.
>> Many discussions recently about evolving the Flink runtime (both on the mailing list and at conferences) show the interest in Flink users in the space that Stateful Functions covers.
>> It seems natural that Stateful Functions should closely co-develop with Apache Flink, ideally as part of the project.
>> 
>> There are many details to be discusses, for example whether this should be added to the Flink core repository, or whether we and to create a separate repository
>> for this. But I think we should start discussing this after we have consensus on whether the community wants this contribution.
>> 
>> Really looking forward to hear what you think!
>> 
>> Best Regards,
>> Stephan
>> 
> 
> 
> -- 
> Best, Jingsong Lee

Re: [PROPOSAL] Contribute Stateful Functions to Apache Flink

[Zili Chen <wa...@gmail.com>]

+1 to add Stateful Function to FLINK core repository.

Best,
tison.


Becket Qin <be...@gmail.com> 于2019年10月14日周一 下午4:16写道：

> +1 to adding Stateful Function to Flink. It is a very useful addition to
> the Flink ecosystem.
>
> Given this is essentially a new top-level / first-citizen API of Flink, it
> seems better to have it the Flink core repo. This will also avoid letting
> this important new API to be blocked on potential problems of maintaining
> multiple different repositories.
>
> Thanks,
>
> Jiangjie (Becket) Qin
>
> On Sun, Oct 13, 2019 at 4:48 AM Hequn Cheng <ch...@gmail.com> wrote:
>
>> Hi Stephan,
>>
>> Big +1 for adding this to Apache Flink!
>>
>> As for the problem of whether this should be added to the Flink main
>> repository, from my side, I prefer to put it in the main repository. Not
>> only Stateful Functions shares very close relations with the current Flink,
>> but also other libs or modules in Flink can make use of it the other way
>> round in the future. At that time the Flink API stack would also be changed
>> a bit and this would be cool.
>>
>> Best, Hequn
>>
>> On Sat, Oct 12, 2019 at 9:16 PM Biao Liu <mm...@gmail.com> wrote:
>>
>>> Hi Stehpan,
>>>
>>> +1 for having Stateful Functions in Flink.
>>>
>>> Before discussing which repository it should belong, I was wondering if
>>> we have reached an agreement of "splitting flink repository" as Piotr
>>> mentioned or not. It seems that it's just no more further discussion.
>>> It's OK for me to add it to core repository. After all almost everything
>>> is in core repository now. But if we decide to split the core repository
>>> someday, I tend to create a separate repository for Stateful Functions. It
>>> might be good time to take the first step of splitting.
>>>
>>> Thanks,
>>> Biao /'bɪ.aʊ/
>>>
>>>
>>>
>>> On Sat, 12 Oct 2019 at 19:31, Yu Li <ca...@gmail.com> wrote:
>>>
>>>> Hi Stephan,
>>>>
>>>> Big +1 for adding stateful functions to Flink. I believe a lot of user
>>>> would be interested to try this out and I could imagine how this could
>>>> contribute to reduce the TCO for business requiring both streaming
>>>> processing and stateful functions.
>>>>
>>>> And my 2 cents is to put it into flink core repository since I could
>>>> see a tight connection between this library and flink state.
>>>>
>>>> Best Regards,
>>>> Yu
>>>>
>>>>
>>>> On Sat, 12 Oct 2019 at 17:31, jincheng sun <su...@gmail.com>
>>>> wrote:
>>>>
>>>>> Hi Stephan,
>>>>>
>>>>> bit +1 for adding this great features to Apache Flink.
>>>>>
>>>>> Regarding where we should place it, put it into Flink core repository
>>>>> or create a separate repository? I prefer put it into main repository and
>>>>> looking forward the more detail discussion for this decision.
>>>>>
>>>>> Best,
>>>>> Jincheng
>>>>>
>>>>>
>>>>> Jingsong Li <ji...@gmail.com> 于2019年10月12日周六 上午11:32写道：
>>>>>
>>>>>> Hi Stephan,
>>>>>>
>>>>>> big +1 for this contribution. It provides another user interface that
>>>>>> is easy to use and popular at this time. these functions, It's hard for
>>>>>> users to write in SQL/TableApi, while using DataStream is too complex.
>>>>>> (We've done some stateFun kind jobs using DataStream before). With
>>>>>> statefun, it is very easy.
>>>>>>
>>>>>> I think it's also a good opportunity to exercise Flink's core
>>>>>> capabilities. I looked at stateful-functions-flink briefly, it is very
>>>>>> interesting. I think there are many other things Flink can improve. So I
>>>>>> think it's a better thing to put it into Flink, and the improvement for it
>>>>>> will be more natural in the future.
>>>>>>
>>>>>> Best,
>>>>>> Jingsong Lee
>>>>>>
>>>>>> On Fri, Oct 11, 2019 at 7:33 PM Dawid Wysakowicz <
>>>>>> dwysakowicz@apache.org> wrote:
>>>>>>
>>>>>>> Hi Stephan,
>>>>>>>
>>>>>>> I think this is a nice library, but what I like more about it is
>>>>>>> that it suggests exploring different use-cases. I think it definitely makes
>>>>>>> sense for the Flink community to explore more lightweight applications that
>>>>>>> reuses resources. Therefore I definitely think it is a good idea for Flink
>>>>>>> community to accept this contribution and help maintaining it.
>>>>>>>
>>>>>>> Personally I'd prefer to have it in a separate repository. There
>>>>>>> were a few discussions before where different people were suggesting to
>>>>>>> extract connectors and other libraries to separate repositories. Moreover I
>>>>>>> think it could serve as an example for the Flink ecosystem website[1]. This
>>>>>>> could be the first project in there and give a good impression that the
>>>>>>> community sees potential in the ecosystem website.
>>>>>>>
>>>>>>> Lastly, I'm wondering if this should go through PMC vote according
>>>>>>> to our bylaws[2]. In the end the suggestion is to adopt an existing code
>>>>>>> base as is. It also proposes a new programs concept that could result in a
>>>>>>> shift of priorities for the community in a long run.
>>>>>>>
>>>>>>> Best,
>>>>>>>
>>>>>>> Dawid
>>>>>>>
>>>>>>> [1]
>>>>>>> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Create-a-Flink-ecosystem-website-td27519.html
>>>>>>>
>>>>>>> [2] https://cwiki.apache.org/confluence/display/FLINK/Flink+Bylaws
>>>>>>> On 11/10/2019 13:12, Till Rohrmann wrote:
>>>>>>>
>>>>>>> Hi Stephan,
>>>>>>>
>>>>>>> +1 for adding stateful functions to Flink. I believe the new set of
>>>>>>> applications this feature will unlock will be super interesting for new and
>>>>>>> existing Flink users alike.
>>>>>>>
>>>>>>> One reason for not including it in the main repository would to not
>>>>>>> being bound to Flink's release cadence. This would allow to release faster
>>>>>>> and more often. However, I believe that having it eventually in Flink's
>>>>>>> main repository would be beneficial in the long run.
>>>>>>>
>>>>>>> Cheers,
>>>>>>> Till
>>>>>>>
>>>>>>> On Fri, Oct 11, 2019 at 12:56 PM Trevor Grant <
>>>>>>> trevor.d.grant@gmail.com> wrote:
>>>>>>>
>>>>>>>> +1 non-binding on contribution.
>>>>>>>>
>>>>>>>> Separate repo, or feature branch to start maybe? I just feel like
>>>>>>>> in the beginning this thing is going to have lots of breaking changes that
>>>>>>>> maybe aren't going to fit well with tests / other "v1+" release code. Just
>>>>>>>> my .02.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Fri, Oct 11, 2019 at 4:38 AM Stephan Ewen <se...@apache.org>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Dear Flink Community!
>>>>>>>>>
>>>>>>>>> Some of you probably heard it already: On Tuesday, at Flink
>>>>>>>>> Forward Berlin, we announced **Stateful Functions**.
>>>>>>>>>
>>>>>>>>> Stateful Functions is a library on Flink to implement general
>>>>>>>>> purpose applications. It is built around stateful functions (who would have
>>>>>>>>> thunk)
>>>>>>>>> that can communicate arbitrarily through messages, have consistent
>>>>>>>>> state, and a small resource footprint. They are a bit like keyed
>>>>>>>>> ProcessFunctions
>>>>>>>>> that can send each other messages.
>>>>>>>>> As simple as this sounds, this means you can now communicate in
>>>>>>>>> non-DAG patterns, so it allows users to build programs they cannot build
>>>>>>>>> with Flink.
>>>>>>>>> It also has other neat properties, like multiplexing of functions,
>>>>>>>>> modular composition, tooling both container-based deployments and
>>>>>>>>> as-a-Flink-job deployments.
>>>>>>>>>
>>>>>>>>> You can find out more about it here
>>>>>>>>>   - Website: https://statefun.io/
>>>>>>>>>   - Code: https://github.com/ververica/stateful-functions
>>>>>>>>>   - Talk with motivation:
>>>>>>>>> https://speakerdeck.com/stephanewen/stateful-functions-building-general-purpose-applications-and-services-on-apache-flink?slide=12
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Now for the main issue: **We would like to contribute this project
>>>>>>>>> to Apache Flink**
>>>>>>>>>
>>>>>>>>> I believe that this is a great fit for both sides.
>>>>>>>>> For the Flink community, it would be a way to extend the
>>>>>>>>> capabilities and use cases of Flink into a completely different type of
>>>>>>>>> applications and thus grow the community into this new field.
>>>>>>>>> Many discussions recently about evolving the Flink runtime (both
>>>>>>>>> on the mailing list and at conferences) show the interest in Flink users in
>>>>>>>>> the space that Stateful Functions covers.
>>>>>>>>> It seems natural that Stateful Functions should closely co-develop
>>>>>>>>> with Apache Flink, ideally as part of the project.
>>>>>>>>>
>>>>>>>>> There are many details to be discusses, for example whether this
>>>>>>>>> should be added to the Flink core repository, or whether we and to create a
>>>>>>>>> separate repository
>>>>>>>>> for this. But I think we should start discussing this after we
>>>>>>>>> have consensus on whether the community wants this contribution.
>>>>>>>>>
>>>>>>>>> Really looking forward to hear what you think!
>>>>>>>>>
>>>>>>>>> Best Regards,
>>>>>>>>> Stephan
>>>>>>>>>
>>>>>>>>>
>>>>>>
>>>>>> --
>>>>>> Best, Jingsong Lee
>>>>>>
>>>>>

Re: [PROPOSAL] Contribute Stateful Functions to Apache Flink

[Flavio Pompermaier <po...@okkam.it>]

I'm obviously pro about promoting the usage of this amazing library but,
maybe, in this early stage I'd try to keep it as a separate project.
However, this really depends about how frequently the code is goin to
change..the Flink main repo is becoming more and more complex to handle due
to the increasing number of open issues and PRs.
I think this library is a perfect fit to test an alternative build system
that is based on multiple git repositories.

Best,
Flavio

Il Lun 14 Ott 2019, 14:53 vino yang <ya...@gmail.com> ha scritto:

> +1 to add Stateful Function to flink core to let it stay in the Flink
> repository.
>
> Best,
> Vino
>
> Stephan Ewen <se...@apache.org> 于2019年10月14日周一 下午7:29写道：
>
> > Thank you all for the encouraging feedback! So far the reaction to add
> this
> > to Flink was exclusively positive, which is really great to see!
> >
> > To make this happen, here would be the next steps:
> >
> > (1) As per the bylaws, a contribution like that would need a PMC vote,
> > because it is a commitment to take this and shepherd
> > it in the future. I will kick that off next.
> >
> > (2) The biggest open question in the current discussion would be whether
> to
> > go with a separate repository, or put it into Flink core.
> > Related to the repository discussion is also how to link and present this
> > on the Flink website.
> > I will spin off a separate discussion for that, to keep the threads
> > focused.
> >
> > Best,
> > Stephan
> >
> >
> > On Mon, Oct 14, 2019 at 10:16 AM Becket Qin <be...@gmail.com>
> wrote:
> >
> > > +1 to adding Stateful Function to Flink. It is a very useful addition
> to
> > > the Flink ecosystem.
> > >
> > > Given this is essentially a new top-level / first-citizen API of Flink,
> > it
> > > seems better to have it the Flink core repo. This will also avoid
> letting
> > > this important new API to be blocked on potential problems of
> maintaining
> > > multiple different repositories.
> > >
> > > Thanks,
> > >
> > > Jiangjie (Becket) Qin
> > >
> > > On Sun, Oct 13, 2019 at 4:48 AM Hequn Cheng <ch...@gmail.com>
> > wrote:
> > >
> > >> Hi Stephan,
> > >>
> > >> Big +1 for adding this to Apache Flink!
> > >>
> > >> As for the problem of whether this should be added to the Flink main
> > >> repository, from my side, I prefer to put it in the main repository.
> Not
> > >> only Stateful Functions shares very close relations with the current
> > Flink,
> > >> but also other libs or modules in Flink can make use of it the other
> way
> > >> round in the future. At that time the Flink API stack would also be
> > changed
> > >> a bit and this would be cool.
> > >>
> > >> Best, Hequn
> > >>
> > >> On Sat, Oct 12, 2019 at 9:16 PM Biao Liu <mm...@gmail.com> wrote:
> > >>
> > >>> Hi Stehpan,
> > >>>
> > >>> +1 for having Stateful Functions in Flink.
> > >>>
> > >>> Before discussing which repository it should belong, I was wondering
> if
> > >>> we have reached an agreement of "splitting flink repository" as Piotr
> > >>> mentioned or not. It seems that it's just no more further discussion.
> > >>> It's OK for me to add it to core repository. After all almost
> > everything
> > >>> is in core repository now. But if we decide to split the core
> > repository
> > >>> someday, I tend to create a separate repository for Stateful
> > Functions. It
> > >>> might be good time to take the first step of splitting.
> > >>>
> > >>> Thanks,
> > >>> Biao /'bɪ.aʊ/
> > >>>
> > >>>
> > >>>
> > >>> On Sat, 12 Oct 2019 at 19:31, Yu Li <ca...@gmail.com> wrote:
> > >>>
> > >>>> Hi Stephan,
> > >>>>
> > >>>> Big +1 for adding stateful functions to Flink. I believe a lot of
> user
> > >>>> would be interested to try this out and I could imagine how this
> could
> > >>>> contribute to reduce the TCO for business requiring both streaming
> > >>>> processing and stateful functions.
> > >>>>
> > >>>> And my 2 cents is to put it into flink core repository since I could
> > >>>> see a tight connection between this library and flink state.
> > >>>>
> > >>>> Best Regards,
> > >>>> Yu
> > >>>>
> > >>>>
> > >>>> On Sat, 12 Oct 2019 at 17:31, jincheng sun <
> sunjincheng121@gmail.com>
> > >>>> wrote:
> > >>>>
> > >>>>> Hi Stephan,
> > >>>>>
> > >>>>> bit +1 for adding this great features to Apache Flink.
> > >>>>>
> > >>>>> Regarding where we should place it, put it into Flink core
> repository
> > >>>>> or create a separate repository? I prefer put it into main
> > repository and
> > >>>>> looking forward the more detail discussion for this decision.
> > >>>>>
> > >>>>> Best,
> > >>>>> Jincheng
> > >>>>>
> > >>>>>
> > >>>>> Jingsong Li <ji...@gmail.com> 于2019年10月12日周六 上午11:32写道：
> > >>>>>
> > >>>>>> Hi Stephan,
> > >>>>>>
> > >>>>>> big +1 for this contribution. It provides another user interface
> > that
> > >>>>>> is easy to use and popular at this time. these functions, It's
> hard
> > for
> > >>>>>> users to write in SQL/TableApi, while using DataStream is too
> > complex.
> > >>>>>> (We've done some stateFun kind jobs using DataStream before). With
> > >>>>>> statefun, it is very easy.
> > >>>>>>
> > >>>>>> I think it's also a good opportunity to exercise Flink's core
> > >>>>>> capabilities. I looked at stateful-functions-flink briefly, it is
> > very
> > >>>>>> interesting. I think there are many other things Flink can
> improve.
> > So I
> > >>>>>> think it's a better thing to put it into Flink, and the
> improvement
> > for it
> > >>>>>> will be more natural in the future.
> > >>>>>>
> > >>>>>> Best,
> > >>>>>> Jingsong Lee
> > >>>>>>
> > >>>>>> On Fri, Oct 11, 2019 at 7:33 PM Dawid Wysakowicz <
> > >>>>>> dwysakowicz@apache.org> wrote:
> > >>>>>>
> > >>>>>>> Hi Stephan,
> > >>>>>>>
> > >>>>>>> I think this is a nice library, but what I like more about it is
> > >>>>>>> that it suggests exploring different use-cases. I think it
> > definitely makes
> > >>>>>>> sense for the Flink community to explore more lightweight
> > applications that
> > >>>>>>> reuses resources. Therefore I definitely think it is a good idea
> > for Flink
> > >>>>>>> community to accept this contribution and help maintaining it.
> > >>>>>>>
> > >>>>>>> Personally I'd prefer to have it in a separate repository. There
> > >>>>>>> were a few discussions before where different people were
> > suggesting to
> > >>>>>>> extract connectors and other libraries to separate repositories.
> > Moreover I
> > >>>>>>> think it could serve as an example for the Flink ecosystem
> > website[1]. This
> > >>>>>>> could be the first project in there and give a good impression
> > that the
> > >>>>>>> community sees potential in the ecosystem website.
> > >>>>>>>
> > >>>>>>> Lastly, I'm wondering if this should go through PMC vote
> according
> > >>>>>>> to our bylaws[2]. In the end the suggestion is to adopt an
> > existing code
> > >>>>>>> base as is. It also proposes a new programs concept that could
> > result in a
> > >>>>>>> shift of priorities for the community in a long run.
> > >>>>>>>
> > >>>>>>> Best,
> > >>>>>>>
> > >>>>>>> Dawid
> > >>>>>>>
> > >>>>>>> [1]
> > >>>>>>>
> >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Create-a-Flink-ecosystem-website-td27519.html
> > >>>>>>>
> > >>>>>>> [2]
> https://cwiki.apache.org/confluence/display/FLINK/Flink+Bylaws
> > >>>>>>> On 11/10/2019 13:12, Till Rohrmann wrote:
> > >>>>>>>
> > >>>>>>> Hi Stephan,
> > >>>>>>>
> > >>>>>>> +1 for adding stateful functions to Flink. I believe the new set
> of
> > >>>>>>> applications this feature will unlock will be super interesting
> > for new and
> > >>>>>>> existing Flink users alike.
> > >>>>>>>
> > >>>>>>> One reason for not including it in the main repository would to
> not
> > >>>>>>> being bound to Flink's release cadence. This would allow to
> > release faster
> > >>>>>>> and more often. However, I believe that having it eventually in
> > Flink's
> > >>>>>>> main repository would be beneficial in the long run.
> > >>>>>>>
> > >>>>>>> Cheers,
> > >>>>>>> Till
> > >>>>>>>
> > >>>>>>> On Fri, Oct 11, 2019 at 12:56 PM Trevor Grant <
> > >>>>>>> trevor.d.grant@gmail.com> wrote:
> > >>>>>>>
> > >>>>>>>> +1 non-binding on contribution.
> > >>>>>>>>
> > >>>>>>>> Separate repo, or feature branch to start maybe? I just feel
> like
> > >>>>>>>> in the beginning this thing is going to have lots of breaking
> > changes that
> > >>>>>>>> maybe aren't going to fit well with tests / other "v1+" release
> > code. Just
> > >>>>>>>> my .02.
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>> On Fri, Oct 11, 2019 at 4:38 AM Stephan Ewen <se...@apache.org>
> > >>>>>>>> wrote:
> > >>>>>>>>
> > >>>>>>>>> Dear Flink Community!
> > >>>>>>>>>
> > >>>>>>>>> Some of you probably heard it already: On Tuesday, at Flink
> > >>>>>>>>> Forward Berlin, we announced **Stateful Functions**.
> > >>>>>>>>>
> > >>>>>>>>> Stateful Functions is a library on Flink to implement general
> > >>>>>>>>> purpose applications. It is built around stateful functions
> (who
> > would have
> > >>>>>>>>> thunk)
> > >>>>>>>>> that can communicate arbitrarily through messages, have
> > consistent
> > >>>>>>>>> state, and a small resource footprint. They are a bit like
> keyed
> > >>>>>>>>> ProcessFunctions
> > >>>>>>>>> that can send each other messages.
> > >>>>>>>>> As simple as this sounds, this means you can now communicate in
> > >>>>>>>>> non-DAG patterns, so it allows users to build programs they
> > cannot build
> > >>>>>>>>> with Flink.
> > >>>>>>>>> It also has other neat properties, like multiplexing of
> > functions,
> > >>>>>>>>> modular composition, tooling both container-based deployments
> and
> > >>>>>>>>> as-a-Flink-job deployments.
> > >>>>>>>>>
> > >>>>>>>>> You can find out more about it here
> > >>>>>>>>>   - Website: https://statefun.io/
> > >>>>>>>>>   - Code: https://github.com/ververica/stateful-functions
> > >>>>>>>>>   - Talk with motivation:
> > >>>>>>>>>
> >
> https://speakerdeck.com/stephanewen/stateful-functions-building-general-purpose-applications-and-services-on-apache-flink?slide=12
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>> Now for the main issue: **We would like to contribute this
> > project
> > >>>>>>>>> to Apache Flink**
> > >>>>>>>>>
> > >>>>>>>>> I believe that this is a great fit for both sides.
> > >>>>>>>>> For the Flink community, it would be a way to extend the
> > >>>>>>>>> capabilities and use cases of Flink into a completely different
> > type of
> > >>>>>>>>> applications and thus grow the community into this new field.
> > >>>>>>>>> Many discussions recently about evolving the Flink runtime
> (both
> > >>>>>>>>> on the mailing list and at conferences) show the interest in
> > Flink users in
> > >>>>>>>>> the space that Stateful Functions covers.
> > >>>>>>>>> It seems natural that Stateful Functions should closely
> > co-develop
> > >>>>>>>>> with Apache Flink, ideally as part of the project.
> > >>>>>>>>>
> > >>>>>>>>> There are many details to be discusses, for example whether
> this
> > >>>>>>>>> should be added to the Flink core repository, or whether we and
> > to create a
> > >>>>>>>>> separate repository
> > >>>>>>>>> for this. But I think we should start discussing this after we
> > >>>>>>>>> have consensus on whether the community wants this
> contribution.
> > >>>>>>>>>
> > >>>>>>>>> Really looking forward to hear what you think!
> > >>>>>>>>>
> > >>>>>>>>> Best Regards,
> > >>>>>>>>> Stephan
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>
> > >>>>>> --
> > >>>>>> Best, Jingsong Lee
> > >>>>>>
> > >>>>>
> >
>

Re: [PROPOSAL] Contribute Stateful Functions to Apache Flink

[vino yang <ya...@gmail.com>]

+1 to add Stateful Function to flink core to let it stay in the Flink
repository.

Best,
Vino

Stephan Ewen <se...@apache.org> 于2019年10月14日周一 下午7:29写道：

> Thank you all for the encouraging feedback! So far the reaction to add this
> to Flink was exclusively positive, which is really great to see!
>
> To make this happen, here would be the next steps:
>
> (1) As per the bylaws, a contribution like that would need a PMC vote,
> because it is a commitment to take this and shepherd
> it in the future. I will kick that off next.
>
> (2) The biggest open question in the current discussion would be whether to
> go with a separate repository, or put it into Flink core.
> Related to the repository discussion is also how to link and present this
> on the Flink website.
> I will spin off a separate discussion for that, to keep the threads
> focused.
>
> Best,
> Stephan
>
>
> On Mon, Oct 14, 2019 at 10:16 AM Becket Qin <be...@gmail.com> wrote:
>
> > +1 to adding Stateful Function to Flink. It is a very useful addition to
> > the Flink ecosystem.
> >
> > Given this is essentially a new top-level / first-citizen API of Flink,
> it
> > seems better to have it the Flink core repo. This will also avoid letting
> > this important new API to be blocked on potential problems of maintaining
> > multiple different repositories.
> >
> > Thanks,
> >
> > Jiangjie (Becket) Qin
> >
> > On Sun, Oct 13, 2019 at 4:48 AM Hequn Cheng <ch...@gmail.com>
> wrote:
> >
> >> Hi Stephan,
> >>
> >> Big +1 for adding this to Apache Flink!
> >>
> >> As for the problem of whether this should be added to the Flink main
> >> repository, from my side, I prefer to put it in the main repository. Not
> >> only Stateful Functions shares very close relations with the current
> Flink,
> >> but also other libs or modules in Flink can make use of it the other way
> >> round in the future. At that time the Flink API stack would also be
> changed
> >> a bit and this would be cool.
> >>
> >> Best, Hequn
> >>
> >> On Sat, Oct 12, 2019 at 9:16 PM Biao Liu <mm...@gmail.com> wrote:
> >>
> >>> Hi Stehpan,
> >>>
> >>> +1 for having Stateful Functions in Flink.
> >>>
> >>> Before discussing which repository it should belong, I was wondering if
> >>> we have reached an agreement of "splitting flink repository" as Piotr
> >>> mentioned or not. It seems that it's just no more further discussion.
> >>> It's OK for me to add it to core repository. After all almost
> everything
> >>> is in core repository now. But if we decide to split the core
> repository
> >>> someday, I tend to create a separate repository for Stateful
> Functions. It
> >>> might be good time to take the first step of splitting.
> >>>
> >>> Thanks,
> >>> Biao /'bɪ.aʊ/
> >>>
> >>>
> >>>
> >>> On Sat, 12 Oct 2019 at 19:31, Yu Li <ca...@gmail.com> wrote:
> >>>
> >>>> Hi Stephan,
> >>>>
> >>>> Big +1 for adding stateful functions to Flink. I believe a lot of user
> >>>> would be interested to try this out and I could imagine how this could
> >>>> contribute to reduce the TCO for business requiring both streaming
> >>>> processing and stateful functions.
> >>>>
> >>>> And my 2 cents is to put it into flink core repository since I could
> >>>> see a tight connection between this library and flink state.
> >>>>
> >>>> Best Regards,
> >>>> Yu
> >>>>
> >>>>
> >>>> On Sat, 12 Oct 2019 at 17:31, jincheng sun <su...@gmail.com>
> >>>> wrote:
> >>>>
> >>>>> Hi Stephan,
> >>>>>
> >>>>> bit +1 for adding this great features to Apache Flink.
> >>>>>
> >>>>> Regarding where we should place it, put it into Flink core repository
> >>>>> or create a separate repository? I prefer put it into main
> repository and
> >>>>> looking forward the more detail discussion for this decision.
> >>>>>
> >>>>> Best,
> >>>>> Jincheng
> >>>>>
> >>>>>
> >>>>> Jingsong Li <ji...@gmail.com> 于2019年10月12日周六 上午11:32写道：
> >>>>>
> >>>>>> Hi Stephan,
> >>>>>>
> >>>>>> big +1 for this contribution. It provides another user interface
> that
> >>>>>> is easy to use and popular at this time. these functions, It's hard
> for
> >>>>>> users to write in SQL/TableApi, while using DataStream is too
> complex.
> >>>>>> (We've done some stateFun kind jobs using DataStream before). With
> >>>>>> statefun, it is very easy.
> >>>>>>
> >>>>>> I think it's also a good opportunity to exercise Flink's core
> >>>>>> capabilities. I looked at stateful-functions-flink briefly, it is
> very
> >>>>>> interesting. I think there are many other things Flink can improve.
> So I
> >>>>>> think it's a better thing to put it into Flink, and the improvement
> for it
> >>>>>> will be more natural in the future.
> >>>>>>
> >>>>>> Best,
> >>>>>> Jingsong Lee
> >>>>>>
> >>>>>> On Fri, Oct 11, 2019 at 7:33 PM Dawid Wysakowicz <
> >>>>>> dwysakowicz@apache.org> wrote:
> >>>>>>
> >>>>>>> Hi Stephan,
> >>>>>>>
> >>>>>>> I think this is a nice library, but what I like more about it is
> >>>>>>> that it suggests exploring different use-cases. I think it
> definitely makes
> >>>>>>> sense for the Flink community to explore more lightweight
> applications that
> >>>>>>> reuses resources. Therefore I definitely think it is a good idea
> for Flink
> >>>>>>> community to accept this contribution and help maintaining it.
> >>>>>>>
> >>>>>>> Personally I'd prefer to have it in a separate repository. There
> >>>>>>> were a few discussions before where different people were
> suggesting to
> >>>>>>> extract connectors and other libraries to separate repositories.
> Moreover I
> >>>>>>> think it could serve as an example for the Flink ecosystem
> website[1]. This
> >>>>>>> could be the first project in there and give a good impression
> that the
> >>>>>>> community sees potential in the ecosystem website.
> >>>>>>>
> >>>>>>> Lastly, I'm wondering if this should go through PMC vote according
> >>>>>>> to our bylaws[2]. In the end the suggestion is to adopt an
> existing code
> >>>>>>> base as is. It also proposes a new programs concept that could
> result in a
> >>>>>>> shift of priorities for the community in a long run.
> >>>>>>>
> >>>>>>> Best,
> >>>>>>>
> >>>>>>> Dawid
> >>>>>>>
> >>>>>>> [1]
> >>>>>>>
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Create-a-Flink-ecosystem-website-td27519.html
> >>>>>>>
> >>>>>>> [2] https://cwiki.apache.org/confluence/display/FLINK/Flink+Bylaws
> >>>>>>> On 11/10/2019 13:12, Till Rohrmann wrote:
> >>>>>>>
> >>>>>>> Hi Stephan,
> >>>>>>>
> >>>>>>> +1 for adding stateful functions to Flink. I believe the new set of
> >>>>>>> applications this feature will unlock will be super interesting
> for new and
> >>>>>>> existing Flink users alike.
> >>>>>>>
> >>>>>>> One reason for not including it in the main repository would to not
> >>>>>>> being bound to Flink's release cadence. This would allow to
> release faster
> >>>>>>> and more often. However, I believe that having it eventually in
> Flink's
> >>>>>>> main repository would be beneficial in the long run.
> >>>>>>>
> >>>>>>> Cheers,
> >>>>>>> Till
> >>>>>>>
> >>>>>>> On Fri, Oct 11, 2019 at 12:56 PM Trevor Grant <
> >>>>>>> trevor.d.grant@gmail.com> wrote:
> >>>>>>>
> >>>>>>>> +1 non-binding on contribution.
> >>>>>>>>
> >>>>>>>> Separate repo, or feature branch to start maybe? I just feel like
> >>>>>>>> in the beginning this thing is going to have lots of breaking
> changes that
> >>>>>>>> maybe aren't going to fit well with tests / other "v1+" release
> code. Just
> >>>>>>>> my .02.
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> On Fri, Oct 11, 2019 at 4:38 AM Stephan Ewen <se...@apache.org>
> >>>>>>>> wrote:
> >>>>>>>>
> >>>>>>>>> Dear Flink Community!
> >>>>>>>>>
> >>>>>>>>> Some of you probably heard it already: On Tuesday, at Flink
> >>>>>>>>> Forward Berlin, we announced **Stateful Functions**.
> >>>>>>>>>
> >>>>>>>>> Stateful Functions is a library on Flink to implement general
> >>>>>>>>> purpose applications. It is built around stateful functions (who
> would have
> >>>>>>>>> thunk)
> >>>>>>>>> that can communicate arbitrarily through messages, have
> consistent
> >>>>>>>>> state, and a small resource footprint. They are a bit like keyed
> >>>>>>>>> ProcessFunctions
> >>>>>>>>> that can send each other messages.
> >>>>>>>>> As simple as this sounds, this means you can now communicate in
> >>>>>>>>> non-DAG patterns, so it allows users to build programs they
> cannot build
> >>>>>>>>> with Flink.
> >>>>>>>>> It also has other neat properties, like multiplexing of
> functions,
> >>>>>>>>> modular composition, tooling both container-based deployments and
> >>>>>>>>> as-a-Flink-job deployments.
> >>>>>>>>>
> >>>>>>>>> You can find out more about it here
> >>>>>>>>>   - Website: https://statefun.io/
> >>>>>>>>>   - Code: https://github.com/ververica/stateful-functions
> >>>>>>>>>   - Talk with motivation:
> >>>>>>>>>
> https://speakerdeck.com/stephanewen/stateful-functions-building-general-purpose-applications-and-services-on-apache-flink?slide=12
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> Now for the main issue: **We would like to contribute this
> project
> >>>>>>>>> to Apache Flink**
> >>>>>>>>>
> >>>>>>>>> I believe that this is a great fit for both sides.
> >>>>>>>>> For the Flink community, it would be a way to extend the
> >>>>>>>>> capabilities and use cases of Flink into a completely different
> type of
> >>>>>>>>> applications and thus grow the community into this new field.
> >>>>>>>>> Many discussions recently about evolving the Flink runtime (both
> >>>>>>>>> on the mailing list and at conferences) show the interest in
> Flink users in
> >>>>>>>>> the space that Stateful Functions covers.
> >>>>>>>>> It seems natural that Stateful Functions should closely
> co-develop
> >>>>>>>>> with Apache Flink, ideally as part of the project.
> >>>>>>>>>
> >>>>>>>>> There are many details to be discusses, for example whether this
> >>>>>>>>> should be added to the Flink core repository, or whether we and
> to create a
> >>>>>>>>> separate repository
> >>>>>>>>> for this. But I think we should start discussing this after we
> >>>>>>>>> have consensus on whether the community wants this contribution.
> >>>>>>>>>
> >>>>>>>>> Really looking forward to hear what you think!
> >>>>>>>>>
> >>>>>>>>> Best Regards,
> >>>>>>>>> Stephan
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>
> >>>>>> --
> >>>>>> Best, Jingsong Lee
> >>>>>>
> >>>>>
>

Re: [PROPOSAL] Contribute Stateful Functions to Apache Flink

[Stephan Ewen <se...@apache.org>]

Thank you all for the encouraging feedback! So far the reaction to add this
to Flink was exclusively positive, which is really great to see!

To make this happen, here would be the next steps:

(1) As per the bylaws, a contribution like that would need a PMC vote,
because it is a commitment to take this and shepherd
it in the future. I will kick that off next.

(2) The biggest open question in the current discussion would be whether to
go with a separate repository, or put it into Flink core.
Related to the repository discussion is also how to link and present this
on the Flink website.
I will spin off a separate discussion for that, to keep the threads focused.

Best,
Stephan


On Mon, Oct 14, 2019 at 10:16 AM Becket Qin <be...@gmail.com> wrote:

> +1 to adding Stateful Function to Flink. It is a very useful addition to
> the Flink ecosystem.
>
> Given this is essentially a new top-level / first-citizen API of Flink, it
> seems better to have it the Flink core repo. This will also avoid letting
> this important new API to be blocked on potential problems of maintaining
> multiple different repositories.
>
> Thanks,
>
> Jiangjie (Becket) Qin
>
> On Sun, Oct 13, 2019 at 4:48 AM Hequn Cheng <ch...@gmail.com> wrote:
>
>> Hi Stephan,
>>
>> Big +1 for adding this to Apache Flink!
>>
>> As for the problem of whether this should be added to the Flink main
>> repository, from my side, I prefer to put it in the main repository. Not
>> only Stateful Functions shares very close relations with the current Flink,
>> but also other libs or modules in Flink can make use of it the other way
>> round in the future. At that time the Flink API stack would also be changed
>> a bit and this would be cool.
>>
>> Best, Hequn
>>
>> On Sat, Oct 12, 2019 at 9:16 PM Biao Liu <mm...@gmail.com> wrote:
>>
>>> Hi Stehpan,
>>>
>>> +1 for having Stateful Functions in Flink.
>>>
>>> Before discussing which repository it should belong, I was wondering if
>>> we have reached an agreement of "splitting flink repository" as Piotr
>>> mentioned or not. It seems that it's just no more further discussion.
>>> It's OK for me to add it to core repository. After all almost everything
>>> is in core repository now. But if we decide to split the core repository
>>> someday, I tend to create a separate repository for Stateful Functions. It
>>> might be good time to take the first step of splitting.
>>>
>>> Thanks,
>>> Biao /'bɪ.aʊ/
>>>
>>>
>>>
>>> On Sat, 12 Oct 2019 at 19:31, Yu Li <ca...@gmail.com> wrote:
>>>
>>>> Hi Stephan,
>>>>
>>>> Big +1 for adding stateful functions to Flink. I believe a lot of user
>>>> would be interested to try this out and I could imagine how this could
>>>> contribute to reduce the TCO for business requiring both streaming
>>>> processing and stateful functions.
>>>>
>>>> And my 2 cents is to put it into flink core repository since I could
>>>> see a tight connection between this library and flink state.
>>>>
>>>> Best Regards,
>>>> Yu
>>>>
>>>>
>>>> On Sat, 12 Oct 2019 at 17:31, jincheng sun <su...@gmail.com>
>>>> wrote:
>>>>
>>>>> Hi Stephan,
>>>>>
>>>>> bit +1 for adding this great features to Apache Flink.
>>>>>
>>>>> Regarding where we should place it, put it into Flink core repository
>>>>> or create a separate repository? I prefer put it into main repository and
>>>>> looking forward the more detail discussion for this decision.
>>>>>
>>>>> Best,
>>>>> Jincheng
>>>>>
>>>>>
>>>>> Jingsong Li <ji...@gmail.com> 于2019年10月12日周六 上午11:32写道：
>>>>>
>>>>>> Hi Stephan,
>>>>>>
>>>>>> big +1 for this contribution. It provides another user interface that
>>>>>> is easy to use and popular at this time. these functions, It's hard for
>>>>>> users to write in SQL/TableApi, while using DataStream is too complex.
>>>>>> (We've done some stateFun kind jobs using DataStream before). With
>>>>>> statefun, it is very easy.
>>>>>>
>>>>>> I think it's also a good opportunity to exercise Flink's core
>>>>>> capabilities. I looked at stateful-functions-flink briefly, it is very
>>>>>> interesting. I think there are many other things Flink can improve. So I
>>>>>> think it's a better thing to put it into Flink, and the improvement for it
>>>>>> will be more natural in the future.
>>>>>>
>>>>>> Best,
>>>>>> Jingsong Lee
>>>>>>
>>>>>> On Fri, Oct 11, 2019 at 7:33 PM Dawid Wysakowicz <
>>>>>> dwysakowicz@apache.org> wrote:
>>>>>>
>>>>>>> Hi Stephan,
>>>>>>>
>>>>>>> I think this is a nice library, but what I like more about it is
>>>>>>> that it suggests exploring different use-cases. I think it definitely makes
>>>>>>> sense for the Flink community to explore more lightweight applications that
>>>>>>> reuses resources. Therefore I definitely think it is a good idea for Flink
>>>>>>> community to accept this contribution and help maintaining it.
>>>>>>>
>>>>>>> Personally I'd prefer to have it in a separate repository. There
>>>>>>> were a few discussions before where different people were suggesting to
>>>>>>> extract connectors and other libraries to separate repositories. Moreover I
>>>>>>> think it could serve as an example for the Flink ecosystem website[1]. This
>>>>>>> could be the first project in there and give a good impression that the
>>>>>>> community sees potential in the ecosystem website.
>>>>>>>
>>>>>>> Lastly, I'm wondering if this should go through PMC vote according
>>>>>>> to our bylaws[2]. In the end the suggestion is to adopt an existing code
>>>>>>> base as is. It also proposes a new programs concept that could result in a
>>>>>>> shift of priorities for the community in a long run.
>>>>>>>
>>>>>>> Best,
>>>>>>>
>>>>>>> Dawid
>>>>>>>
>>>>>>> [1]
>>>>>>> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Create-a-Flink-ecosystem-website-td27519.html
>>>>>>>
>>>>>>> [2] https://cwiki.apache.org/confluence/display/FLINK/Flink+Bylaws
>>>>>>> On 11/10/2019 13:12, Till Rohrmann wrote:
>>>>>>>
>>>>>>> Hi Stephan,
>>>>>>>
>>>>>>> +1 for adding stateful functions to Flink. I believe the new set of
>>>>>>> applications this feature will unlock will be super interesting for new and
>>>>>>> existing Flink users alike.
>>>>>>>
>>>>>>> One reason for not including it in the main repository would to not
>>>>>>> being bound to Flink's release cadence. This would allow to release faster
>>>>>>> and more often. However, I believe that having it eventually in Flink's
>>>>>>> main repository would be beneficial in the long run.
>>>>>>>
>>>>>>> Cheers,
>>>>>>> Till
>>>>>>>
>>>>>>> On Fri, Oct 11, 2019 at 12:56 PM Trevor Grant <
>>>>>>> trevor.d.grant@gmail.com> wrote:
>>>>>>>
>>>>>>>> +1 non-binding on contribution.
>>>>>>>>
>>>>>>>> Separate repo, or feature branch to start maybe? I just feel like
>>>>>>>> in the beginning this thing is going to have lots of breaking changes that
>>>>>>>> maybe aren't going to fit well with tests / other "v1+" release code. Just
>>>>>>>> my .02.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Fri, Oct 11, 2019 at 4:38 AM Stephan Ewen <se...@apache.org>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Dear Flink Community!
>>>>>>>>>
>>>>>>>>> Some of you probably heard it already: On Tuesday, at Flink
>>>>>>>>> Forward Berlin, we announced **Stateful Functions**.
>>>>>>>>>
>>>>>>>>> Stateful Functions is a library on Flink to implement general
>>>>>>>>> purpose applications. It is built around stateful functions (who would have
>>>>>>>>> thunk)
>>>>>>>>> that can communicate arbitrarily through messages, have consistent
>>>>>>>>> state, and a small resource footprint. They are a bit like keyed
>>>>>>>>> ProcessFunctions
>>>>>>>>> that can send each other messages.
>>>>>>>>> As simple as this sounds, this means you can now communicate in
>>>>>>>>> non-DAG patterns, so it allows users to build programs they cannot build
>>>>>>>>> with Flink.
>>>>>>>>> It also has other neat properties, like multiplexing of functions,
>>>>>>>>> modular composition, tooling both container-based deployments and
>>>>>>>>> as-a-Flink-job deployments.
>>>>>>>>>
>>>>>>>>> You can find out more about it here
>>>>>>>>>   - Website: https://statefun.io/
>>>>>>>>>   - Code: https://github.com/ververica/stateful-functions
>>>>>>>>>   - Talk with motivation:
>>>>>>>>> https://speakerdeck.com/stephanewen/stateful-functions-building-general-purpose-applications-and-services-on-apache-flink?slide=12
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Now for the main issue: **We would like to contribute this project
>>>>>>>>> to Apache Flink**
>>>>>>>>>
>>>>>>>>> I believe that this is a great fit for both sides.
>>>>>>>>> For the Flink community, it would be a way to extend the
>>>>>>>>> capabilities and use cases of Flink into a completely different type of
>>>>>>>>> applications and thus grow the community into this new field.
>>>>>>>>> Many discussions recently about evolving the Flink runtime (both
>>>>>>>>> on the mailing list and at conferences) show the interest in Flink users in
>>>>>>>>> the space that Stateful Functions covers.
>>>>>>>>> It seems natural that Stateful Functions should closely co-develop
>>>>>>>>> with Apache Flink, ideally as part of the project.
>>>>>>>>>
>>>>>>>>> There are many details to be discusses, for example whether this
>>>>>>>>> should be added to the Flink core repository, or whether we and to create a
>>>>>>>>> separate repository
>>>>>>>>> for this. But I think we should start discussing this after we
>>>>>>>>> have consensus on whether the community wants this contribution.
>>>>>>>>>
>>>>>>>>> Really looking forward to hear what you think!
>>>>>>>>>
>>>>>>>>> Best Regards,
>>>>>>>>> Stephan
>>>>>>>>>
>>>>>>>>>
>>>>>>
>>>>>> --
>>>>>> Best, Jingsong Lee
>>>>>>
>>>>>

Re: [PROPOSAL] Contribute Stateful Functions to Apache Flink

[Stephan Ewen <se...@apache.org>]

Thank you all for the encouraging feedback! So far the reaction to add this
to Flink was exclusively positive, which is really great to see!

To make this happen, here would be the next steps:

(1) As per the bylaws, a contribution like that would need a PMC vote,
because it is a commitment to take this and shepherd
it in the future. I will kick that off next.

(2) The biggest open question in the current discussion would be whether to
go with a separate repository, or put it into Flink core.
Related to the repository discussion is also how to link and present this
on the Flink website.
I will spin off a separate discussion for that, to keep the threads focused.

Best,
Stephan


On Mon, Oct 14, 2019 at 10:16 AM Becket Qin <be...@gmail.com> wrote:

> +1 to adding Stateful Function to Flink. It is a very useful addition to
> the Flink ecosystem.
>
> Given this is essentially a new top-level / first-citizen API of Flink, it
> seems better to have it the Flink core repo. This will also avoid letting
> this important new API to be blocked on potential problems of maintaining
> multiple different repositories.
>
> Thanks,
>
> Jiangjie (Becket) Qin
>
> On Sun, Oct 13, 2019 at 4:48 AM Hequn Cheng <ch...@gmail.com> wrote:
>
>> Hi Stephan,
>>
>> Big +1 for adding this to Apache Flink!
>>
>> As for the problem of whether this should be added to the Flink main
>> repository, from my side, I prefer to put it in the main repository. Not
>> only Stateful Functions shares very close relations with the current Flink,
>> but also other libs or modules in Flink can make use of it the other way
>> round in the future. At that time the Flink API stack would also be changed
>> a bit and this would be cool.
>>
>> Best, Hequn
>>
>> On Sat, Oct 12, 2019 at 9:16 PM Biao Liu <mm...@gmail.com> wrote:
>>
>>> Hi Stehpan,
>>>
>>> +1 for having Stateful Functions in Flink.
>>>
>>> Before discussing which repository it should belong, I was wondering if
>>> we have reached an agreement of "splitting flink repository" as Piotr
>>> mentioned or not. It seems that it's just no more further discussion.
>>> It's OK for me to add it to core repository. After all almost everything
>>> is in core repository now. But if we decide to split the core repository
>>> someday, I tend to create a separate repository for Stateful Functions. It
>>> might be good time to take the first step of splitting.
>>>
>>> Thanks,
>>> Biao /'bɪ.aʊ/
>>>
>>>
>>>
>>> On Sat, 12 Oct 2019 at 19:31, Yu Li <ca...@gmail.com> wrote:
>>>
>>>> Hi Stephan,
>>>>
>>>> Big +1 for adding stateful functions to Flink. I believe a lot of user
>>>> would be interested to try this out and I could imagine how this could
>>>> contribute to reduce the TCO for business requiring both streaming
>>>> processing and stateful functions.
>>>>
>>>> And my 2 cents is to put it into flink core repository since I could
>>>> see a tight connection between this library and flink state.
>>>>
>>>> Best Regards,
>>>> Yu
>>>>
>>>>
>>>> On Sat, 12 Oct 2019 at 17:31, jincheng sun <su...@gmail.com>
>>>> wrote:
>>>>
>>>>> Hi Stephan,
>>>>>
>>>>> bit +1 for adding this great features to Apache Flink.
>>>>>
>>>>> Regarding where we should place it, put it into Flink core repository
>>>>> or create a separate repository? I prefer put it into main repository and
>>>>> looking forward the more detail discussion for this decision.
>>>>>
>>>>> Best,
>>>>> Jincheng
>>>>>
>>>>>
>>>>> Jingsong Li <ji...@gmail.com> 于2019年10月12日周六 上午11:32写道：
>>>>>
>>>>>> Hi Stephan,
>>>>>>
>>>>>> big +1 for this contribution. It provides another user interface that
>>>>>> is easy to use and popular at this time. these functions, It's hard for
>>>>>> users to write in SQL/TableApi, while using DataStream is too complex.
>>>>>> (We've done some stateFun kind jobs using DataStream before). With
>>>>>> statefun, it is very easy.
>>>>>>
>>>>>> I think it's also a good opportunity to exercise Flink's core
>>>>>> capabilities. I looked at stateful-functions-flink briefly, it is very
>>>>>> interesting. I think there are many other things Flink can improve. So I
>>>>>> think it's a better thing to put it into Flink, and the improvement for it
>>>>>> will be more natural in the future.
>>>>>>
>>>>>> Best,
>>>>>> Jingsong Lee
>>>>>>
>>>>>> On Fri, Oct 11, 2019 at 7:33 PM Dawid Wysakowicz <
>>>>>> dwysakowicz@apache.org> wrote:
>>>>>>
>>>>>>> Hi Stephan,
>>>>>>>
>>>>>>> I think this is a nice library, but what I like more about it is
>>>>>>> that it suggests exploring different use-cases. I think it definitely makes
>>>>>>> sense for the Flink community to explore more lightweight applications that
>>>>>>> reuses resources. Therefore I definitely think it is a good idea for Flink
>>>>>>> community to accept this contribution and help maintaining it.
>>>>>>>
>>>>>>> Personally I'd prefer to have it in a separate repository. There
>>>>>>> were a few discussions before where different people were suggesting to
>>>>>>> extract connectors and other libraries to separate repositories. Moreover I
>>>>>>> think it could serve as an example for the Flink ecosystem website[1]. This
>>>>>>> could be the first project in there and give a good impression that the
>>>>>>> community sees potential in the ecosystem website.
>>>>>>>
>>>>>>> Lastly, I'm wondering if this should go through PMC vote according
>>>>>>> to our bylaws[2]. In the end the suggestion is to adopt an existing code
>>>>>>> base as is. It also proposes a new programs concept that could result in a
>>>>>>> shift of priorities for the community in a long run.
>>>>>>>
>>>>>>> Best,
>>>>>>>
>>>>>>> Dawid
>>>>>>>
>>>>>>> [1]
>>>>>>> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Create-a-Flink-ecosystem-website-td27519.html
>>>>>>>
>>>>>>> [2] https://cwiki.apache.org/confluence/display/FLINK/Flink+Bylaws
>>>>>>> On 11/10/2019 13:12, Till Rohrmann wrote:
>>>>>>>
>>>>>>> Hi Stephan,
>>>>>>>
>>>>>>> +1 for adding stateful functions to Flink. I believe the new set of
>>>>>>> applications this feature will unlock will be super interesting for new and
>>>>>>> existing Flink users alike.
>>>>>>>
>>>>>>> One reason for not including it in the main repository would to not
>>>>>>> being bound to Flink's release cadence. This would allow to release faster
>>>>>>> and more often. However, I believe that having it eventually in Flink's
>>>>>>> main repository would be beneficial in the long run.
>>>>>>>
>>>>>>> Cheers,
>>>>>>> Till
>>>>>>>
>>>>>>> On Fri, Oct 11, 2019 at 12:56 PM Trevor Grant <
>>>>>>> trevor.d.grant@gmail.com> wrote:
>>>>>>>
>>>>>>>> +1 non-binding on contribution.
>>>>>>>>
>>>>>>>> Separate repo, or feature branch to start maybe? I just feel like
>>>>>>>> in the beginning this thing is going to have lots of breaking changes that
>>>>>>>> maybe aren't going to fit well with tests / other "v1+" release code. Just
>>>>>>>> my .02.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Fri, Oct 11, 2019 at 4:38 AM Stephan Ewen <se...@apache.org>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Dear Flink Community!
>>>>>>>>>
>>>>>>>>> Some of you probably heard it already: On Tuesday, at Flink
>>>>>>>>> Forward Berlin, we announced **Stateful Functions**.
>>>>>>>>>
>>>>>>>>> Stateful Functions is a library on Flink to implement general
>>>>>>>>> purpose applications. It is built around stateful functions (who would have
>>>>>>>>> thunk)
>>>>>>>>> that can communicate arbitrarily through messages, have consistent
>>>>>>>>> state, and a small resource footprint. They are a bit like keyed
>>>>>>>>> ProcessFunctions
>>>>>>>>> that can send each other messages.
>>>>>>>>> As simple as this sounds, this means you can now communicate in
>>>>>>>>> non-DAG patterns, so it allows users to build programs they cannot build
>>>>>>>>> with Flink.
>>>>>>>>> It also has other neat properties, like multiplexing of functions,
>>>>>>>>> modular composition, tooling both container-based deployments and
>>>>>>>>> as-a-Flink-job deployments.
>>>>>>>>>
>>>>>>>>> You can find out more about it here
>>>>>>>>>   - Website: https://statefun.io/
>>>>>>>>>   - Code: https://github.com/ververica/stateful-functions
>>>>>>>>>   - Talk with motivation:
>>>>>>>>> https://speakerdeck.com/stephanewen/stateful-functions-building-general-purpose-applications-and-services-on-apache-flink?slide=12
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Now for the main issue: **We would like to contribute this project
>>>>>>>>> to Apache Flink**
>>>>>>>>>
>>>>>>>>> I believe that this is a great fit for both sides.
>>>>>>>>> For the Flink community, it would be a way to extend the
>>>>>>>>> capabilities and use cases of Flink into a completely different type of
>>>>>>>>> applications and thus grow the community into this new field.
>>>>>>>>> Many discussions recently about evolving the Flink runtime (both
>>>>>>>>> on the mailing list and at conferences) show the interest in Flink users in
>>>>>>>>> the space that Stateful Functions covers.
>>>>>>>>> It seems natural that Stateful Functions should closely co-develop
>>>>>>>>> with Apache Flink, ideally as part of the project.
>>>>>>>>>
>>>>>>>>> There are many details to be discusses, for example whether this
>>>>>>>>> should be added to the Flink core repository, or whether we and to create a
>>>>>>>>> separate repository
>>>>>>>>> for this. But I think we should start discussing this after we
>>>>>>>>> have consensus on whether the community wants this contribution.
>>>>>>>>>
>>>>>>>>> Really looking forward to hear what you think!
>>>>>>>>>
>>>>>>>>> Best Regards,
>>>>>>>>> Stephan
>>>>>>>>>
>>>>>>>>>
>>>>>>
>>>>>> --
>>>>>> Best, Jingsong Lee
>>>>>>
>>>>>

Re: [PROPOSAL] Contribute Stateful Functions to Apache Flink

[Zili Chen <wa...@gmail.com>]

+1 to add Stateful Function to FLINK core repository.

Best,
tison.


Becket Qin <be...@gmail.com> 于2019年10月14日周一 下午4:16写道：

> +1 to adding Stateful Function to Flink. It is a very useful addition to
> the Flink ecosystem.
>
> Given this is essentially a new top-level / first-citizen API of Flink, it
> seems better to have it the Flink core repo. This will also avoid letting
> this important new API to be blocked on potential problems of maintaining
> multiple different repositories.
>
> Thanks,
>
> Jiangjie (Becket) Qin
>
> On Sun, Oct 13, 2019 at 4:48 AM Hequn Cheng <ch...@gmail.com> wrote:
>
>> Hi Stephan,
>>
>> Big +1 for adding this to Apache Flink!
>>
>> As for the problem of whether this should be added to the Flink main
>> repository, from my side, I prefer to put it in the main repository. Not
>> only Stateful Functions shares very close relations with the current Flink,
>> but also other libs or modules in Flink can make use of it the other way
>> round in the future. At that time the Flink API stack would also be changed
>> a bit and this would be cool.
>>
>> Best, Hequn
>>
>> On Sat, Oct 12, 2019 at 9:16 PM Biao Liu <mm...@gmail.com> wrote:
>>
>>> Hi Stehpan,
>>>
>>> +1 for having Stateful Functions in Flink.
>>>
>>> Before discussing which repository it should belong, I was wondering if
>>> we have reached an agreement of "splitting flink repository" as Piotr
>>> mentioned or not. It seems that it's just no more further discussion.
>>> It's OK for me to add it to core repository. After all almost everything
>>> is in core repository now. But if we decide to split the core repository
>>> someday, I tend to create a separate repository for Stateful Functions. It
>>> might be good time to take the first step of splitting.
>>>
>>> Thanks,
>>> Biao /'bɪ.aʊ/
>>>
>>>
>>>
>>> On Sat, 12 Oct 2019 at 19:31, Yu Li <ca...@gmail.com> wrote:
>>>
>>>> Hi Stephan,
>>>>
>>>> Big +1 for adding stateful functions to Flink. I believe a lot of user
>>>> would be interested to try this out and I could imagine how this could
>>>> contribute to reduce the TCO for business requiring both streaming
>>>> processing and stateful functions.
>>>>
>>>> And my 2 cents is to put it into flink core repository since I could
>>>> see a tight connection between this library and flink state.
>>>>
>>>> Best Regards,
>>>> Yu
>>>>
>>>>
>>>> On Sat, 12 Oct 2019 at 17:31, jincheng sun <su...@gmail.com>
>>>> wrote:
>>>>
>>>>> Hi Stephan,
>>>>>
>>>>> bit +1 for adding this great features to Apache Flink.
>>>>>
>>>>> Regarding where we should place it, put it into Flink core repository
>>>>> or create a separate repository? I prefer put it into main repository and
>>>>> looking forward the more detail discussion for this decision.
>>>>>
>>>>> Best,
>>>>> Jincheng
>>>>>
>>>>>
>>>>> Jingsong Li <ji...@gmail.com> 于2019年10月12日周六 上午11:32写道：
>>>>>
>>>>>> Hi Stephan,
>>>>>>
>>>>>> big +1 for this contribution. It provides another user interface that
>>>>>> is easy to use and popular at this time. these functions, It's hard for
>>>>>> users to write in SQL/TableApi, while using DataStream is too complex.
>>>>>> (We've done some stateFun kind jobs using DataStream before). With
>>>>>> statefun, it is very easy.
>>>>>>
>>>>>> I think it's also a good opportunity to exercise Flink's core
>>>>>> capabilities. I looked at stateful-functions-flink briefly, it is very
>>>>>> interesting. I think there are many other things Flink can improve. So I
>>>>>> think it's a better thing to put it into Flink, and the improvement for it
>>>>>> will be more natural in the future.
>>>>>>
>>>>>> Best,
>>>>>> Jingsong Lee
>>>>>>
>>>>>> On Fri, Oct 11, 2019 at 7:33 PM Dawid Wysakowicz <
>>>>>> dwysakowicz@apache.org> wrote:
>>>>>>
>>>>>>> Hi Stephan,
>>>>>>>
>>>>>>> I think this is a nice library, but what I like more about it is
>>>>>>> that it suggests exploring different use-cases. I think it definitely makes
>>>>>>> sense for the Flink community to explore more lightweight applications that
>>>>>>> reuses resources. Therefore I definitely think it is a good idea for Flink
>>>>>>> community to accept this contribution and help maintaining it.
>>>>>>>
>>>>>>> Personally I'd prefer to have it in a separate repository. There
>>>>>>> were a few discussions before where different people were suggesting to
>>>>>>> extract connectors and other libraries to separate repositories. Moreover I
>>>>>>> think it could serve as an example for the Flink ecosystem website[1]. This
>>>>>>> could be the first project in there and give a good impression that the
>>>>>>> community sees potential in the ecosystem website.
>>>>>>>
>>>>>>> Lastly, I'm wondering if this should go through PMC vote according
>>>>>>> to our bylaws[2]. In the end the suggestion is to adopt an existing code
>>>>>>> base as is. It also proposes a new programs concept that could result in a
>>>>>>> shift of priorities for the community in a long run.
>>>>>>>
>>>>>>> Best,
>>>>>>>
>>>>>>> Dawid
>>>>>>>
>>>>>>> [1]
>>>>>>> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Create-a-Flink-ecosystem-website-td27519.html
>>>>>>>
>>>>>>> [2] https://cwiki.apache.org/confluence/display/FLINK/Flink+Bylaws
>>>>>>> On 11/10/2019 13:12, Till Rohrmann wrote:
>>>>>>>
>>>>>>> Hi Stephan,
>>>>>>>
>>>>>>> +1 for adding stateful functions to Flink. I believe the new set of
>>>>>>> applications this feature will unlock will be super interesting for new and
>>>>>>> existing Flink users alike.
>>>>>>>
>>>>>>> One reason for not including it in the main repository would to not
>>>>>>> being bound to Flink's release cadence. This would allow to release faster
>>>>>>> and more often. However, I believe that having it eventually in Flink's
>>>>>>> main repository would be beneficial in the long run.
>>>>>>>
>>>>>>> Cheers,
>>>>>>> Till
>>>>>>>
>>>>>>> On Fri, Oct 11, 2019 at 12:56 PM Trevor Grant <
>>>>>>> trevor.d.grant@gmail.com> wrote:
>>>>>>>
>>>>>>>> +1 non-binding on contribution.
>>>>>>>>
>>>>>>>> Separate repo, or feature branch to start maybe? I just feel like
>>>>>>>> in the beginning this thing is going to have lots of breaking changes that
>>>>>>>> maybe aren't going to fit well with tests / other "v1+" release code. Just
>>>>>>>> my .02.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Fri, Oct 11, 2019 at 4:38 AM Stephan Ewen <se...@apache.org>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Dear Flink Community!
>>>>>>>>>
>>>>>>>>> Some of you probably heard it already: On Tuesday, at Flink
>>>>>>>>> Forward Berlin, we announced **Stateful Functions**.
>>>>>>>>>
>>>>>>>>> Stateful Functions is a library on Flink to implement general
>>>>>>>>> purpose applications. It is built around stateful functions (who would have
>>>>>>>>> thunk)
>>>>>>>>> that can communicate arbitrarily through messages, have consistent
>>>>>>>>> state, and a small resource footprint. They are a bit like keyed
>>>>>>>>> ProcessFunctions
>>>>>>>>> that can send each other messages.
>>>>>>>>> As simple as this sounds, this means you can now communicate in
>>>>>>>>> non-DAG patterns, so it allows users to build programs they cannot build
>>>>>>>>> with Flink.
>>>>>>>>> It also has other neat properties, like multiplexing of functions,
>>>>>>>>> modular composition, tooling both container-based deployments and
>>>>>>>>> as-a-Flink-job deployments.
>>>>>>>>>
>>>>>>>>> You can find out more about it here
>>>>>>>>>   - Website: https://statefun.io/
>>>>>>>>>   - Code: https://github.com/ververica/stateful-functions
>>>>>>>>>   - Talk with motivation:
>>>>>>>>> https://speakerdeck.com/stephanewen/stateful-functions-building-general-purpose-applications-and-services-on-apache-flink?slide=12
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Now for the main issue: **We would like to contribute this project
>>>>>>>>> to Apache Flink**
>>>>>>>>>
>>>>>>>>> I believe that this is a great fit for both sides.
>>>>>>>>> For the Flink community, it would be a way to extend the
>>>>>>>>> capabilities and use cases of Flink into a completely different type of
>>>>>>>>> applications and thus grow the community into this new field.
>>>>>>>>> Many discussions recently about evolving the Flink runtime (both
>>>>>>>>> on the mailing list and at conferences) show the interest in Flink users in
>>>>>>>>> the space that Stateful Functions covers.
>>>>>>>>> It seems natural that Stateful Functions should closely co-develop
>>>>>>>>> with Apache Flink, ideally as part of the project.
>>>>>>>>>
>>>>>>>>> There are many details to be discusses, for example whether this
>>>>>>>>> should be added to the Flink core repository, or whether we and to create a
>>>>>>>>> separate repository
>>>>>>>>> for this. But I think we should start discussing this after we
>>>>>>>>> have consensus on whether the community wants this contribution.
>>>>>>>>>
>>>>>>>>> Really looking forward to hear what you think!
>>>>>>>>>
>>>>>>>>> Best Regards,
>>>>>>>>> Stephan
>>>>>>>>>
>>>>>>>>>
>>>>>>
>>>>>> --
>>>>>> Best, Jingsong Lee
>>>>>>
>>>>>

Re: [PROPOSAL] Contribute Stateful Functions to Apache Flink

[Becket Qin <be...@gmail.com>]

+1 to adding Stateful Function to Flink. It is a very useful addition to
the Flink ecosystem.

Given this is essentially a new top-level / first-citizen API of Flink, it
seems better to have it the Flink core repo. This will also avoid letting
this important new API to be blocked on potential problems of maintaining
multiple different repositories.

Thanks,

Jiangjie (Becket) Qin

On Sun, Oct 13, 2019 at 4:48 AM Hequn Cheng <ch...@gmail.com> wrote:

> Hi Stephan,
>
> Big +1 for adding this to Apache Flink!
>
> As for the problem of whether this should be added to the Flink main
> repository, from my side, I prefer to put it in the main repository. Not
> only Stateful Functions shares very close relations with the current Flink,
> but also other libs or modules in Flink can make use of it the other way
> round in the future. At that time the Flink API stack would also be changed
> a bit and this would be cool.
>
> Best, Hequn
>
> On Sat, Oct 12, 2019 at 9:16 PM Biao Liu <mm...@gmail.com> wrote:
>
>> Hi Stehpan,
>>
>> +1 for having Stateful Functions in Flink.
>>
>> Before discussing which repository it should belong, I was wondering if
>> we have reached an agreement of "splitting flink repository" as Piotr
>> mentioned or not. It seems that it's just no more further discussion.
>> It's OK for me to add it to core repository. After all almost everything
>> is in core repository now. But if we decide to split the core repository
>> someday, I tend to create a separate repository for Stateful Functions. It
>> might be good time to take the first step of splitting.
>>
>> Thanks,
>> Biao /'bɪ.aʊ/
>>
>>
>>
>> On Sat, 12 Oct 2019 at 19:31, Yu Li <ca...@gmail.com> wrote:
>>
>>> Hi Stephan,
>>>
>>> Big +1 for adding stateful functions to Flink. I believe a lot of user
>>> would be interested to try this out and I could imagine how this could
>>> contribute to reduce the TCO for business requiring both streaming
>>> processing and stateful functions.
>>>
>>> And my 2 cents is to put it into flink core repository since I could see
>>> a tight connection between this library and flink state.
>>>
>>> Best Regards,
>>> Yu
>>>
>>>
>>> On Sat, 12 Oct 2019 at 17:31, jincheng sun <su...@gmail.com>
>>> wrote:
>>>
>>>> Hi Stephan,
>>>>
>>>> bit +1 for adding this great features to Apache Flink.
>>>>
>>>> Regarding where we should place it, put it into Flink core repository
>>>> or create a separate repository? I prefer put it into main repository and
>>>> looking forward the more detail discussion for this decision.
>>>>
>>>> Best,
>>>> Jincheng
>>>>
>>>>
>>>> Jingsong Li <ji...@gmail.com> 于2019年10月12日周六 上午11:32写道：
>>>>
>>>>> Hi Stephan,
>>>>>
>>>>> big +1 for this contribution. It provides another user interface that
>>>>> is easy to use and popular at this time. these functions, It's hard for
>>>>> users to write in SQL/TableApi, while using DataStream is too complex.
>>>>> (We've done some stateFun kind jobs using DataStream before). With
>>>>> statefun, it is very easy.
>>>>>
>>>>> I think it's also a good opportunity to exercise Flink's core
>>>>> capabilities. I looked at stateful-functions-flink briefly, it is very
>>>>> interesting. I think there are many other things Flink can improve. So I
>>>>> think it's a better thing to put it into Flink, and the improvement for it
>>>>> will be more natural in the future.
>>>>>
>>>>> Best,
>>>>> Jingsong Lee
>>>>>
>>>>> On Fri, Oct 11, 2019 at 7:33 PM Dawid Wysakowicz <
>>>>> dwysakowicz@apache.org> wrote:
>>>>>
>>>>>> Hi Stephan,
>>>>>>
>>>>>> I think this is a nice library, but what I like more about it is that
>>>>>> it suggests exploring different use-cases. I think it definitely makes
>>>>>> sense for the Flink community to explore more lightweight applications that
>>>>>> reuses resources. Therefore I definitely think it is a good idea for Flink
>>>>>> community to accept this contribution and help maintaining it.
>>>>>>
>>>>>> Personally I'd prefer to have it in a separate repository. There were
>>>>>> a few discussions before where different people were suggesting to extract
>>>>>> connectors and other libraries to separate repositories. Moreover I think
>>>>>> it could serve as an example for the Flink ecosystem website[1]. This could
>>>>>> be the first project in there and give a good impression that the community
>>>>>> sees potential in the ecosystem website.
>>>>>>
>>>>>> Lastly, I'm wondering if this should go through PMC vote according to
>>>>>> our bylaws[2]. In the end the suggestion is to adopt an existing code base
>>>>>> as is. It also proposes a new programs concept that could result in a shift
>>>>>> of priorities for the community in a long run.
>>>>>>
>>>>>> Best,
>>>>>>
>>>>>> Dawid
>>>>>>
>>>>>> [1]
>>>>>> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Create-a-Flink-ecosystem-website-td27519.html
>>>>>>
>>>>>> [2] https://cwiki.apache.org/confluence/display/FLINK/Flink+Bylaws
>>>>>> On 11/10/2019 13:12, Till Rohrmann wrote:
>>>>>>
>>>>>> Hi Stephan,
>>>>>>
>>>>>> +1 for adding stateful functions to Flink. I believe the new set of
>>>>>> applications this feature will unlock will be super interesting for new and
>>>>>> existing Flink users alike.
>>>>>>
>>>>>> One reason for not including it in the main repository would to not
>>>>>> being bound to Flink's release cadence. This would allow to release faster
>>>>>> and more often. However, I believe that having it eventually in Flink's
>>>>>> main repository would be beneficial in the long run.
>>>>>>
>>>>>> Cheers,
>>>>>> Till
>>>>>>
>>>>>> On Fri, Oct 11, 2019 at 12:56 PM Trevor Grant <
>>>>>> trevor.d.grant@gmail.com> wrote:
>>>>>>
>>>>>>> +1 non-binding on contribution.
>>>>>>>
>>>>>>> Separate repo, or feature branch to start maybe? I just feel like in
>>>>>>> the beginning this thing is going to have lots of breaking changes that
>>>>>>> maybe aren't going to fit well with tests / other "v1+" release code. Just
>>>>>>> my .02.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Fri, Oct 11, 2019 at 4:38 AM Stephan Ewen <se...@apache.org>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Dear Flink Community!
>>>>>>>>
>>>>>>>> Some of you probably heard it already: On Tuesday, at Flink Forward
>>>>>>>> Berlin, we announced **Stateful Functions**.
>>>>>>>>
>>>>>>>> Stateful Functions is a library on Flink to implement general
>>>>>>>> purpose applications. It is built around stateful functions (who would have
>>>>>>>> thunk)
>>>>>>>> that can communicate arbitrarily through messages, have consistent
>>>>>>>> state, and a small resource footprint. They are a bit like keyed
>>>>>>>> ProcessFunctions
>>>>>>>> that can send each other messages.
>>>>>>>> As simple as this sounds, this means you can now communicate in
>>>>>>>> non-DAG patterns, so it allows users to build programs they cannot build
>>>>>>>> with Flink.
>>>>>>>> It also has other neat properties, like multiplexing of functions,
>>>>>>>> modular composition, tooling both container-based deployments and
>>>>>>>> as-a-Flink-job deployments.
>>>>>>>>
>>>>>>>> You can find out more about it here
>>>>>>>>   - Website: https://statefun.io/
>>>>>>>>   - Code: https://github.com/ververica/stateful-functions
>>>>>>>>   - Talk with motivation:
>>>>>>>> https://speakerdeck.com/stephanewen/stateful-functions-building-general-purpose-applications-and-services-on-apache-flink?slide=12
>>>>>>>>
>>>>>>>>
>>>>>>>> Now for the main issue: **We would like to contribute this project
>>>>>>>> to Apache Flink**
>>>>>>>>
>>>>>>>> I believe that this is a great fit for both sides.
>>>>>>>> For the Flink community, it would be a way to extend the
>>>>>>>> capabilities and use cases of Flink into a completely different type of
>>>>>>>> applications and thus grow the community into this new field.
>>>>>>>> Many discussions recently about evolving the Flink runtime (both on
>>>>>>>> the mailing list and at conferences) show the interest in Flink users in
>>>>>>>> the space that Stateful Functions covers.
>>>>>>>> It seems natural that Stateful Functions should closely co-develop
>>>>>>>> with Apache Flink, ideally as part of the project.
>>>>>>>>
>>>>>>>> There are many details to be discusses, for example whether this
>>>>>>>> should be added to the Flink core repository, or whether we and to create a
>>>>>>>> separate repository
>>>>>>>> for this. But I think we should start discussing this after we have
>>>>>>>> consensus on whether the community wants this contribution.
>>>>>>>>
>>>>>>>> Really looking forward to hear what you think!
>>>>>>>>
>>>>>>>> Best Regards,
>>>>>>>> Stephan
>>>>>>>>
>>>>>>>>
>>>>>
>>>>> --
>>>>> Best, Jingsong Lee
>>>>>
>>>>

Re: [PROPOSAL] Contribute Stateful Functions to Apache Flink

[Dian Fu <di...@gmail.com>]

Hi Stephan,

Big +1 for adding stateful functions to Apache Flink! The use cases unlocked with this feature are very interesting and promising.

Regarding to whether to place it into Flink core repository, personally I perfer to put it in the main repository. This feature introduces a new set of APIs and it will support a new set of applications. It enriches the API stack of Apache Flink. This is somewhat simlar to the Table API & SQL, State Processor API, CEP library, etc. If the applications supported by this feature are important enough for Flink, it's more appropriate to put it directly into the main repository.

Regards,
Dian

> 在 2019年10月13日，上午10:47，Hequn Cheng <ch...@gmail.com> 写道：
> 
> Hi Stephan,
> 
> Big +1 for adding this to Apache Flink! 
> 
> As for the problem of whether this should be added to the Flink main repository, from my side, I prefer to put it in the main repository. Not only Stateful Functions shares very close relations with the current Flink, but also other libs or modules in Flink can make use of it the other way round in the future. At that time the Flink API stack would also be changed a bit and this would be cool.
> 
> Best, Hequn
> 
> On Sat, Oct 12, 2019 at 9:16 PM Biao Liu <mmyy1110@gmail.com <ma...@gmail.com>> wrote:
> Hi Stehpan,
> 
> +1 for having Stateful Functions in Flink.
> 
> Before discussing which repository it should belong, I was wondering if we have reached an agreement of "splitting flink repository" as Piotr mentioned or not. It seems that it's just no more further discussion. 
> It's OK for me to add it to core repository. After all almost everything is in core repository now. But if we decide to split the core repository someday, I tend to create a separate repository for Stateful Functions. It might be good time to take the first step of splitting.
> 
> Thanks,
> Biao /'bɪ.aʊ/
> 
> 
> 
> On Sat, 12 Oct 2019 at 19:31, Yu Li <carp84@gmail.com <ma...@gmail.com>> wrote:
> Hi Stephan,
> 
> Big +1 for adding stateful functions to Flink. I believe a lot of user would be interested to try this out and I could imagine how this could contribute to reduce the TCO for business requiring both streaming processing and stateful functions.
> 
> And my 2 cents is to put it into flink core repository since I could see a tight connection between this library and flink state.
> 
> Best Regards,
> Yu
> 
> 
> On Sat, 12 Oct 2019 at 17:31, jincheng sun <sunjincheng121@gmail.com <ma...@gmail.com>> wrote:
> Hi Stephan,
> 
> bit +1 for adding this great features to Apache Flink.
> 
> Regarding where we should place it, put it into Flink core repository or create a separate repository? I prefer put it into main repository and looking forward the more detail discussion for this decision.
> 
> Best,
> Jincheng
> 
> 
> Jingsong Li <jingsonglee0@gmail.com <ma...@gmail.com>> 于2019年10月12日周六 上午11:32写道：
> Hi Stephan,
> 
> big +1 for this contribution. It provides another user interface that is easy to use and popular at this time. these functions, It's hard for users to write in SQL/TableApi, while using DataStream is too complex. (We've done some stateFun kind jobs using DataStream before). With statefun, it is very easy.
> 
> I think it's also a good opportunity to exercise Flink's core capabilities. I looked at stateful-functions-flink briefly, it is very interesting. I think there are many other things Flink can improve. So I think it's a better thing to put it into Flink, and the improvement for it will be more natural in the future.
> 
> Best,
> Jingsong Lee
> 
> On Fri, Oct 11, 2019 at 7:33 PM Dawid Wysakowicz <dwysakowicz@apache.org <ma...@apache.org>> wrote:
> Hi Stephan,
> 
> I think this is a nice library, but what I like more about it is that it suggests exploring different use-cases. I think it definitely makes sense for the Flink community to explore more lightweight applications that reuses resources. Therefore I definitely think it is a good idea for Flink community to accept this contribution and help maintaining it.
> 
> Personally I'd prefer to have it in a separate repository. There were a few discussions before where different people were suggesting to extract connectors and other libraries to separate repositories. Moreover I think it could serve as an example for the Flink ecosystem website[1]. This could be the first project in there and give a good impression that the community sees potential in the ecosystem website.
> Lastly, I'm wondering if this should go through PMC vote according to our bylaws[2]. In the end the suggestion is to adopt an existing code base as is. It also proposes a new programs concept that could result in a shift of priorities for the community in a long run.
> Best,
> 
> Dawid
> [1] http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Create-a-Flink-ecosystem-website-td27519.html <http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Create-a-Flink-ecosystem-website-td27519.html>
> [2] https://cwiki.apache.org/confluence/display/FLINK/Flink+Bylaws <https://cwiki.apache.org/confluence/display/FLINK/Flink+Bylaws>
> On 11/10/2019 13:12, Till Rohrmann wrote:
>> Hi Stephan,
>> 
>> +1 for adding stateful functions to Flink. I believe the new set of applications this feature will unlock will be super interesting for new and existing Flink users alike.
>> 
>> One reason for not including it in the main repository would to not being bound to Flink's release cadence. This would allow to release faster and more often. However, I believe that having it eventually in Flink's main repository would be beneficial in the long run.
>> 
>> Cheers,
>> Till
>> 
>> On Fri, Oct 11, 2019 at 12:56 PM Trevor Grant <trevor.d.grant@gmail.com <ma...@gmail.com>> wrote:
>> +1 non-binding on contribution. 
>> 
>> Separate repo, or feature branch to start maybe? I just feel like in the beginning this thing is going to have lots of breaking changes that maybe aren't going to fit well with tests / other "v1+" release code. Just my .02. 
>> 
>> 
>> 
>> On Fri, Oct 11, 2019 at 4:38 AM Stephan Ewen <sewen@apache.org <ma...@apache.org>> wrote:
>> Dear Flink Community!
>> 
>> Some of you probably heard it already: On Tuesday, at Flink Forward Berlin, we announced **Stateful Functions**.
>> 
>> Stateful Functions is a library on Flink to implement general purpose applications. It is built around stateful functions (who would have thunk)
>> that can communicate arbitrarily through messages, have consistent state, and a small resource footprint. They are a bit like keyed ProcessFunctions
>> that can send each other messages.
>> As simple as this sounds, this means you can now communicate in non-DAG patterns, so it allows users to build programs they cannot build with Flink.
>> It also has other neat properties, like multiplexing of functions, modular composition, tooling both container-based deployments and as-a-Flink-job deployments.
>> 
>> You can find out more about it here
>>   - Website: https://statefun.io/ <https://statefun.io/>
>>   - Code: https://github.com/ververica/stateful-functions <https://github.com/ververica/stateful-functions>
>>   - Talk with motivation: https://speakerdeck.com/stephanewen/stateful-functions-building-general-purpose-applications-and-services-on-apache-flink?slide=12 <https://speakerdeck.com/stephanewen/stateful-functions-building-general-purpose-applications-and-services-on-apache-flink?slide=12>
>> 
>> 
>> Now for the main issue: **We would like to contribute this project to Apache Flink**
>> 
>> I believe that this is a great fit for both sides.
>> For the Flink community, it would be a way to extend the capabilities and use cases of Flink into a completely different type of applications and thus grow the community into this new field.
>> Many discussions recently about evolving the Flink runtime (both on the mailing list and at conferences) show the interest in Flink users in the space that Stateful Functions covers.
>> It seems natural that Stateful Functions should closely co-develop with Apache Flink, ideally as part of the project.
>> 
>> There are many details to be discusses, for example whether this should be added to the Flink core repository, or whether we and to create a separate repository
>> for this. But I think we should start discussing this after we have consensus on whether the community wants this contribution.
>> 
>> Really looking forward to hear what you think!
>> 
>> Best Regards,
>> Stephan
>> 
> 
> 
> -- 
> Best, Jingsong Lee

Re: [PROPOSAL] Contribute Stateful Functions to Apache Flink

[Becket Qin <be...@gmail.com>]

+1 to adding Stateful Function to Flink. It is a very useful addition to
the Flink ecosystem.

Given this is essentially a new top-level / first-citizen API of Flink, it
seems better to have it the Flink core repo. This will also avoid letting
this important new API to be blocked on potential problems of maintaining
multiple different repositories.

Thanks,

Jiangjie (Becket) Qin

On Sun, Oct 13, 2019 at 4:48 AM Hequn Cheng <ch...@gmail.com> wrote:

> Hi Stephan,
>
> Big +1 for adding this to Apache Flink!
>
> As for the problem of whether this should be added to the Flink main
> repository, from my side, I prefer to put it in the main repository. Not
> only Stateful Functions shares very close relations with the current Flink,
> but also other libs or modules in Flink can make use of it the other way
> round in the future. At that time the Flink API stack would also be changed
> a bit and this would be cool.
>
> Best, Hequn
>
> On Sat, Oct 12, 2019 at 9:16 PM Biao Liu <mm...@gmail.com> wrote:
>
>> Hi Stehpan,
>>
>> +1 for having Stateful Functions in Flink.
>>
>> Before discussing which repository it should belong, I was wondering if
>> we have reached an agreement of "splitting flink repository" as Piotr
>> mentioned or not. It seems that it's just no more further discussion.
>> It's OK for me to add it to core repository. After all almost everything
>> is in core repository now. But if we decide to split the core repository
>> someday, I tend to create a separate repository for Stateful Functions. It
>> might be good time to take the first step of splitting.
>>
>> Thanks,
>> Biao /'bɪ.aʊ/
>>
>>
>>
>> On Sat, 12 Oct 2019 at 19:31, Yu Li <ca...@gmail.com> wrote:
>>
>>> Hi Stephan,
>>>
>>> Big +1 for adding stateful functions to Flink. I believe a lot of user
>>> would be interested to try this out and I could imagine how this could
>>> contribute to reduce the TCO for business requiring both streaming
>>> processing and stateful functions.
>>>
>>> And my 2 cents is to put it into flink core repository since I could see
>>> a tight connection between this library and flink state.
>>>
>>> Best Regards,
>>> Yu
>>>
>>>
>>> On Sat, 12 Oct 2019 at 17:31, jincheng sun <su...@gmail.com>
>>> wrote:
>>>
>>>> Hi Stephan,
>>>>
>>>> bit +1 for adding this great features to Apache Flink.
>>>>
>>>> Regarding where we should place it, put it into Flink core repository
>>>> or create a separate repository? I prefer put it into main repository and
>>>> looking forward the more detail discussion for this decision.
>>>>
>>>> Best,
>>>> Jincheng
>>>>
>>>>
>>>> Jingsong Li <ji...@gmail.com> 于2019年10月12日周六 上午11:32写道：
>>>>
>>>>> Hi Stephan,
>>>>>
>>>>> big +1 for this contribution. It provides another user interface that
>>>>> is easy to use and popular at this time. these functions, It's hard for
>>>>> users to write in SQL/TableApi, while using DataStream is too complex.
>>>>> (We've done some stateFun kind jobs using DataStream before). With
>>>>> statefun, it is very easy.
>>>>>
>>>>> I think it's also a good opportunity to exercise Flink's core
>>>>> capabilities. I looked at stateful-functions-flink briefly, it is very
>>>>> interesting. I think there are many other things Flink can improve. So I
>>>>> think it's a better thing to put it into Flink, and the improvement for it
>>>>> will be more natural in the future.
>>>>>
>>>>> Best,
>>>>> Jingsong Lee
>>>>>
>>>>> On Fri, Oct 11, 2019 at 7:33 PM Dawid Wysakowicz <
>>>>> dwysakowicz@apache.org> wrote:
>>>>>
>>>>>> Hi Stephan,
>>>>>>
>>>>>> I think this is a nice library, but what I like more about it is that
>>>>>> it suggests exploring different use-cases. I think it definitely makes
>>>>>> sense for the Flink community to explore more lightweight applications that
>>>>>> reuses resources. Therefore I definitely think it is a good idea for Flink
>>>>>> community to accept this contribution and help maintaining it.
>>>>>>
>>>>>> Personally I'd prefer to have it in a separate repository. There were
>>>>>> a few discussions before where different people were suggesting to extract
>>>>>> connectors and other libraries to separate repositories. Moreover I think
>>>>>> it could serve as an example for the Flink ecosystem website[1]. This could
>>>>>> be the first project in there and give a good impression that the community
>>>>>> sees potential in the ecosystem website.
>>>>>>
>>>>>> Lastly, I'm wondering if this should go through PMC vote according to
>>>>>> our bylaws[2]. In the end the suggestion is to adopt an existing code base
>>>>>> as is. It also proposes a new programs concept that could result in a shift
>>>>>> of priorities for the community in a long run.
>>>>>>
>>>>>> Best,
>>>>>>
>>>>>> Dawid
>>>>>>
>>>>>> [1]
>>>>>> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Create-a-Flink-ecosystem-website-td27519.html
>>>>>>
>>>>>> [2] https://cwiki.apache.org/confluence/display/FLINK/Flink+Bylaws
>>>>>> On 11/10/2019 13:12, Till Rohrmann wrote:
>>>>>>
>>>>>> Hi Stephan,
>>>>>>
>>>>>> +1 for adding stateful functions to Flink. I believe the new set of
>>>>>> applications this feature will unlock will be super interesting for new and
>>>>>> existing Flink users alike.
>>>>>>
>>>>>> One reason for not including it in the main repository would to not
>>>>>> being bound to Flink's release cadence. This would allow to release faster
>>>>>> and more often. However, I believe that having it eventually in Flink's
>>>>>> main repository would be beneficial in the long run.
>>>>>>
>>>>>> Cheers,
>>>>>> Till
>>>>>>
>>>>>> On Fri, Oct 11, 2019 at 12:56 PM Trevor Grant <
>>>>>> trevor.d.grant@gmail.com> wrote:
>>>>>>
>>>>>>> +1 non-binding on contribution.
>>>>>>>
>>>>>>> Separate repo, or feature branch to start maybe? I just feel like in
>>>>>>> the beginning this thing is going to have lots of breaking changes that
>>>>>>> maybe aren't going to fit well with tests / other "v1+" release code. Just
>>>>>>> my .02.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Fri, Oct 11, 2019 at 4:38 AM Stephan Ewen <se...@apache.org>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Dear Flink Community!
>>>>>>>>
>>>>>>>> Some of you probably heard it already: On Tuesday, at Flink Forward
>>>>>>>> Berlin, we announced **Stateful Functions**.
>>>>>>>>
>>>>>>>> Stateful Functions is a library on Flink to implement general
>>>>>>>> purpose applications. It is built around stateful functions (who would have
>>>>>>>> thunk)
>>>>>>>> that can communicate arbitrarily through messages, have consistent
>>>>>>>> state, and a small resource footprint. They are a bit like keyed
>>>>>>>> ProcessFunctions
>>>>>>>> that can send each other messages.
>>>>>>>> As simple as this sounds, this means you can now communicate in
>>>>>>>> non-DAG patterns, so it allows users to build programs they cannot build
>>>>>>>> with Flink.
>>>>>>>> It also has other neat properties, like multiplexing of functions,
>>>>>>>> modular composition, tooling both container-based deployments and
>>>>>>>> as-a-Flink-job deployments.
>>>>>>>>
>>>>>>>> You can find out more about it here
>>>>>>>>   - Website: https://statefun.io/
>>>>>>>>   - Code: https://github.com/ververica/stateful-functions
>>>>>>>>   - Talk with motivation:
>>>>>>>> https://speakerdeck.com/stephanewen/stateful-functions-building-general-purpose-applications-and-services-on-apache-flink?slide=12
>>>>>>>>
>>>>>>>>
>>>>>>>> Now for the main issue: **We would like to contribute this project
>>>>>>>> to Apache Flink**
>>>>>>>>
>>>>>>>> I believe that this is a great fit for both sides.
>>>>>>>> For the Flink community, it would be a way to extend the
>>>>>>>> capabilities and use cases of Flink into a completely different type of
>>>>>>>> applications and thus grow the community into this new field.
>>>>>>>> Many discussions recently about evolving the Flink runtime (both on
>>>>>>>> the mailing list and at conferences) show the interest in Flink users in
>>>>>>>> the space that Stateful Functions covers.
>>>>>>>> It seems natural that Stateful Functions should closely co-develop
>>>>>>>> with Apache Flink, ideally as part of the project.
>>>>>>>>
>>>>>>>> There are many details to be discusses, for example whether this
>>>>>>>> should be added to the Flink core repository, or whether we and to create a
>>>>>>>> separate repository
>>>>>>>> for this. But I think we should start discussing this after we have
>>>>>>>> consensus on whether the community wants this contribution.
>>>>>>>>
>>>>>>>> Really looking forward to hear what you think!
>>>>>>>>
>>>>>>>> Best Regards,
>>>>>>>> Stephan
>>>>>>>>
>>>>>>>>
>>>>>
>>>>> --
>>>>> Best, Jingsong Lee
>>>>>
>>>>

Re: [PROPOSAL] Contribute Stateful Functions to Apache Flink

[Hequn Cheng <ch...@gmail.com>]

Hi Stephan,

Big +1 for adding this to Apache Flink!

As for the problem of whether this should be added to the Flink main
repository, from my side, I prefer to put it in the main repository. Not
only Stateful Functions shares very close relations with the current Flink,
but also other libs or modules in Flink can make use of it the other way
round in the future. At that time the Flink API stack would also be changed
a bit and this would be cool.

Best, Hequn

On Sat, Oct 12, 2019 at 9:16 PM Biao Liu <mm...@gmail.com> wrote:

> Hi Stehpan,
>
> +1 for having Stateful Functions in Flink.
>
> Before discussing which repository it should belong, I was wondering if we
> have reached an agreement of "splitting flink repository" as Piotr
> mentioned or not. It seems that it's just no more further discussion.
> It's OK for me to add it to core repository. After all almost everything
> is in core repository now. But if we decide to split the core repository
> someday, I tend to create a separate repository for Stateful Functions. It
> might be good time to take the first step of splitting.
>
> Thanks,
> Biao /'bɪ.aʊ/
>
>
>
> On Sat, 12 Oct 2019 at 19:31, Yu Li <ca...@gmail.com> wrote:
>
>> Hi Stephan,
>>
>> Big +1 for adding stateful functions to Flink. I believe a lot of user
>> would be interested to try this out and I could imagine how this could
>> contribute to reduce the TCO for business requiring both streaming
>> processing and stateful functions.
>>
>> And my 2 cents is to put it into flink core repository since I could see
>> a tight connection between this library and flink state.
>>
>> Best Regards,
>> Yu
>>
>>
>> On Sat, 12 Oct 2019 at 17:31, jincheng sun <su...@gmail.com>
>> wrote:
>>
>>> Hi Stephan,
>>>
>>> bit +1 for adding this great features to Apache Flink.
>>>
>>> Regarding where we should place it, put it into Flink core repository or
>>> create a separate repository? I prefer put it into main repository and
>>> looking forward the more detail discussion for this decision.
>>>
>>> Best,
>>> Jincheng
>>>
>>>
>>> Jingsong Li <ji...@gmail.com> 于2019年10月12日周六 上午11:32写道：
>>>
>>>> Hi Stephan,
>>>>
>>>> big +1 for this contribution. It provides another user interface that
>>>> is easy to use and popular at this time. these functions, It's hard for
>>>> users to write in SQL/TableApi, while using DataStream is too complex.
>>>> (We've done some stateFun kind jobs using DataStream before). With
>>>> statefun, it is very easy.
>>>>
>>>> I think it's also a good opportunity to exercise Flink's core
>>>> capabilities. I looked at stateful-functions-flink briefly, it is very
>>>> interesting. I think there are many other things Flink can improve. So I
>>>> think it's a better thing to put it into Flink, and the improvement for it
>>>> will be more natural in the future.
>>>>
>>>> Best,
>>>> Jingsong Lee
>>>>
>>>> On Fri, Oct 11, 2019 at 7:33 PM Dawid Wysakowicz <
>>>> dwysakowicz@apache.org> wrote:
>>>>
>>>>> Hi Stephan,
>>>>>
>>>>> I think this is a nice library, but what I like more about it is that
>>>>> it suggests exploring different use-cases. I think it definitely makes
>>>>> sense for the Flink community to explore more lightweight applications that
>>>>> reuses resources. Therefore I definitely think it is a good idea for Flink
>>>>> community to accept this contribution and help maintaining it.
>>>>>
>>>>> Personally I'd prefer to have it in a separate repository. There were
>>>>> a few discussions before where different people were suggesting to extract
>>>>> connectors and other libraries to separate repositories. Moreover I think
>>>>> it could serve as an example for the Flink ecosystem website[1]. This could
>>>>> be the first project in there and give a good impression that the community
>>>>> sees potential in the ecosystem website.
>>>>>
>>>>> Lastly, I'm wondering if this should go through PMC vote according to
>>>>> our bylaws[2]. In the end the suggestion is to adopt an existing code base
>>>>> as is. It also proposes a new programs concept that could result in a shift
>>>>> of priorities for the community in a long run.
>>>>>
>>>>> Best,
>>>>>
>>>>> Dawid
>>>>>
>>>>> [1]
>>>>> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Create-a-Flink-ecosystem-website-td27519.html
>>>>>
>>>>> [2] https://cwiki.apache.org/confluence/display/FLINK/Flink+Bylaws
>>>>> On 11/10/2019 13:12, Till Rohrmann wrote:
>>>>>
>>>>> Hi Stephan,
>>>>>
>>>>> +1 for adding stateful functions to Flink. I believe the new set of
>>>>> applications this feature will unlock will be super interesting for new and
>>>>> existing Flink users alike.
>>>>>
>>>>> One reason for not including it in the main repository would to not
>>>>> being bound to Flink's release cadence. This would allow to release faster
>>>>> and more often. However, I believe that having it eventually in Flink's
>>>>> main repository would be beneficial in the long run.
>>>>>
>>>>> Cheers,
>>>>> Till
>>>>>
>>>>> On Fri, Oct 11, 2019 at 12:56 PM Trevor Grant <
>>>>> trevor.d.grant@gmail.com> wrote:
>>>>>
>>>>>> +1 non-binding on contribution.
>>>>>>
>>>>>> Separate repo, or feature branch to start maybe? I just feel like in
>>>>>> the beginning this thing is going to have lots of breaking changes that
>>>>>> maybe aren't going to fit well with tests / other "v1+" release code. Just
>>>>>> my .02.
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Fri, Oct 11, 2019 at 4:38 AM Stephan Ewen <se...@apache.org>
>>>>>> wrote:
>>>>>>
>>>>>>> Dear Flink Community!
>>>>>>>
>>>>>>> Some of you probably heard it already: On Tuesday, at Flink Forward
>>>>>>> Berlin, we announced **Stateful Functions**.
>>>>>>>
>>>>>>> Stateful Functions is a library on Flink to implement general
>>>>>>> purpose applications. It is built around stateful functions (who would have
>>>>>>> thunk)
>>>>>>> that can communicate arbitrarily through messages, have consistent
>>>>>>> state, and a small resource footprint. They are a bit like keyed
>>>>>>> ProcessFunctions
>>>>>>> that can send each other messages.
>>>>>>> As simple as this sounds, this means you can now communicate in
>>>>>>> non-DAG patterns, so it allows users to build programs they cannot build
>>>>>>> with Flink.
>>>>>>> It also has other neat properties, like multiplexing of functions,
>>>>>>> modular composition, tooling both container-based deployments and
>>>>>>> as-a-Flink-job deployments.
>>>>>>>
>>>>>>> You can find out more about it here
>>>>>>>   - Website: https://statefun.io/
>>>>>>>   - Code: https://github.com/ververica/stateful-functions
>>>>>>>   - Talk with motivation:
>>>>>>> https://speakerdeck.com/stephanewen/stateful-functions-building-general-purpose-applications-and-services-on-apache-flink?slide=12
>>>>>>>
>>>>>>>
>>>>>>> Now for the main issue: **We would like to contribute this project
>>>>>>> to Apache Flink**
>>>>>>>
>>>>>>> I believe that this is a great fit for both sides.
>>>>>>> For the Flink community, it would be a way to extend the
>>>>>>> capabilities and use cases of Flink into a completely different type of
>>>>>>> applications and thus grow the community into this new field.
>>>>>>> Many discussions recently about evolving the Flink runtime (both on
>>>>>>> the mailing list and at conferences) show the interest in Flink users in
>>>>>>> the space that Stateful Functions covers.
>>>>>>> It seems natural that Stateful Functions should closely co-develop
>>>>>>> with Apache Flink, ideally as part of the project.
>>>>>>>
>>>>>>> There are many details to be discusses, for example whether this
>>>>>>> should be added to the Flink core repository, or whether we and to create a
>>>>>>> separate repository
>>>>>>> for this. But I think we should start discussing this after we have
>>>>>>> consensus on whether the community wants this contribution.
>>>>>>>
>>>>>>> Really looking forward to hear what you think!
>>>>>>>
>>>>>>> Best Regards,
>>>>>>> Stephan
>>>>>>>
>>>>>>>
>>>>
>>>> --
>>>> Best, Jingsong Lee
>>>>
>>>

Re: [PROPOSAL] Contribute Stateful Functions to Apache Flink

[Hequn Cheng <ch...@gmail.com>]

Hi Stephan,

Big +1 for adding this to Apache Flink!

As for the problem of whether this should be added to the Flink main
repository, from my side, I prefer to put it in the main repository. Not
only Stateful Functions shares very close relations with the current Flink,
but also other libs or modules in Flink can make use of it the other way
round in the future. At that time the Flink API stack would also be changed
a bit and this would be cool.

Best, Hequn

On Sat, Oct 12, 2019 at 9:16 PM Biao Liu <mm...@gmail.com> wrote:

> Hi Stehpan,
>
> +1 for having Stateful Functions in Flink.
>
> Before discussing which repository it should belong, I was wondering if we
> have reached an agreement of "splitting flink repository" as Piotr
> mentioned or not. It seems that it's just no more further discussion.
> It's OK for me to add it to core repository. After all almost everything
> is in core repository now. But if we decide to split the core repository
> someday, I tend to create a separate repository for Stateful Functions. It
> might be good time to take the first step of splitting.
>
> Thanks,
> Biao /'bɪ.aʊ/
>
>
>
> On Sat, 12 Oct 2019 at 19:31, Yu Li <ca...@gmail.com> wrote:
>
>> Hi Stephan,
>>
>> Big +1 for adding stateful functions to Flink. I believe a lot of user
>> would be interested to try this out and I could imagine how this could
>> contribute to reduce the TCO for business requiring both streaming
>> processing and stateful functions.
>>
>> And my 2 cents is to put it into flink core repository since I could see
>> a tight connection between this library and flink state.
>>
>> Best Regards,
>> Yu
>>
>>
>> On Sat, 12 Oct 2019 at 17:31, jincheng sun <su...@gmail.com>
>> wrote:
>>
>>> Hi Stephan,
>>>
>>> bit +1 for adding this great features to Apache Flink.
>>>
>>> Regarding where we should place it, put it into Flink core repository or
>>> create a separate repository? I prefer put it into main repository and
>>> looking forward the more detail discussion for this decision.
>>>
>>> Best,
>>> Jincheng
>>>
>>>
>>> Jingsong Li <ji...@gmail.com> 于2019年10月12日周六 上午11:32写道：
>>>
>>>> Hi Stephan,
>>>>
>>>> big +1 for this contribution. It provides another user interface that
>>>> is easy to use and popular at this time. these functions, It's hard for
>>>> users to write in SQL/TableApi, while using DataStream is too complex.
>>>> (We've done some stateFun kind jobs using DataStream before). With
>>>> statefun, it is very easy.
>>>>
>>>> I think it's also a good opportunity to exercise Flink's core
>>>> capabilities. I looked at stateful-functions-flink briefly, it is very
>>>> interesting. I think there are many other things Flink can improve. So I
>>>> think it's a better thing to put it into Flink, and the improvement for it
>>>> will be more natural in the future.
>>>>
>>>> Best,
>>>> Jingsong Lee
>>>>
>>>> On Fri, Oct 11, 2019 at 7:33 PM Dawid Wysakowicz <
>>>> dwysakowicz@apache.org> wrote:
>>>>
>>>>> Hi Stephan,
>>>>>
>>>>> I think this is a nice library, but what I like more about it is that
>>>>> it suggests exploring different use-cases. I think it definitely makes
>>>>> sense for the Flink community to explore more lightweight applications that
>>>>> reuses resources. Therefore I definitely think it is a good idea for Flink
>>>>> community to accept this contribution and help maintaining it.
>>>>>
>>>>> Personally I'd prefer to have it in a separate repository. There were
>>>>> a few discussions before where different people were suggesting to extract
>>>>> connectors and other libraries to separate repositories. Moreover I think
>>>>> it could serve as an example for the Flink ecosystem website[1]. This could
>>>>> be the first project in there and give a good impression that the community
>>>>> sees potential in the ecosystem website.
>>>>>
>>>>> Lastly, I'm wondering if this should go through PMC vote according to
>>>>> our bylaws[2]. In the end the suggestion is to adopt an existing code base
>>>>> as is. It also proposes a new programs concept that could result in a shift
>>>>> of priorities for the community in a long run.
>>>>>
>>>>> Best,
>>>>>
>>>>> Dawid
>>>>>
>>>>> [1]
>>>>> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Create-a-Flink-ecosystem-website-td27519.html
>>>>>
>>>>> [2] https://cwiki.apache.org/confluence/display/FLINK/Flink+Bylaws
>>>>> On 11/10/2019 13:12, Till Rohrmann wrote:
>>>>>
>>>>> Hi Stephan,
>>>>>
>>>>> +1 for adding stateful functions to Flink. I believe the new set of
>>>>> applications this feature will unlock will be super interesting for new and
>>>>> existing Flink users alike.
>>>>>
>>>>> One reason for not including it in the main repository would to not
>>>>> being bound to Flink's release cadence. This would allow to release faster
>>>>> and more often. However, I believe that having it eventually in Flink's
>>>>> main repository would be beneficial in the long run.
>>>>>
>>>>> Cheers,
>>>>> Till
>>>>>
>>>>> On Fri, Oct 11, 2019 at 12:56 PM Trevor Grant <
>>>>> trevor.d.grant@gmail.com> wrote:
>>>>>
>>>>>> +1 non-binding on contribution.
>>>>>>
>>>>>> Separate repo, or feature branch to start maybe? I just feel like in
>>>>>> the beginning this thing is going to have lots of breaking changes that
>>>>>> maybe aren't going to fit well with tests / other "v1+" release code. Just
>>>>>> my .02.
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Fri, Oct 11, 2019 at 4:38 AM Stephan Ewen <se...@apache.org>
>>>>>> wrote:
>>>>>>
>>>>>>> Dear Flink Community!
>>>>>>>
>>>>>>> Some of you probably heard it already: On Tuesday, at Flink Forward
>>>>>>> Berlin, we announced **Stateful Functions**.
>>>>>>>
>>>>>>> Stateful Functions is a library on Flink to implement general
>>>>>>> purpose applications. It is built around stateful functions (who would have
>>>>>>> thunk)
>>>>>>> that can communicate arbitrarily through messages, have consistent
>>>>>>> state, and a small resource footprint. They are a bit like keyed
>>>>>>> ProcessFunctions
>>>>>>> that can send each other messages.
>>>>>>> As simple as this sounds, this means you can now communicate in
>>>>>>> non-DAG patterns, so it allows users to build programs they cannot build
>>>>>>> with Flink.
>>>>>>> It also has other neat properties, like multiplexing of functions,
>>>>>>> modular composition, tooling both container-based deployments and
>>>>>>> as-a-Flink-job deployments.
>>>>>>>
>>>>>>> You can find out more about it here
>>>>>>>   - Website: https://statefun.io/
>>>>>>>   - Code: https://github.com/ververica/stateful-functions
>>>>>>>   - Talk with motivation:
>>>>>>> https://speakerdeck.com/stephanewen/stateful-functions-building-general-purpose-applications-and-services-on-apache-flink?slide=12
>>>>>>>
>>>>>>>
>>>>>>> Now for the main issue: **We would like to contribute this project
>>>>>>> to Apache Flink**
>>>>>>>
>>>>>>> I believe that this is a great fit for both sides.
>>>>>>> For the Flink community, it would be a way to extend the
>>>>>>> capabilities and use cases of Flink into a completely different type of
>>>>>>> applications and thus grow the community into this new field.
>>>>>>> Many discussions recently about evolving the Flink runtime (both on
>>>>>>> the mailing list and at conferences) show the interest in Flink users in
>>>>>>> the space that Stateful Functions covers.
>>>>>>> It seems natural that Stateful Functions should closely co-develop
>>>>>>> with Apache Flink, ideally as part of the project.
>>>>>>>
>>>>>>> There are many details to be discusses, for example whether this
>>>>>>> should be added to the Flink core repository, or whether we and to create a
>>>>>>> separate repository
>>>>>>> for this. But I think we should start discussing this after we have
>>>>>>> consensus on whether the community wants this contribution.
>>>>>>>
>>>>>>> Really looking forward to hear what you think!
>>>>>>>
>>>>>>> Best Regards,
>>>>>>> Stephan
>>>>>>>
>>>>>>>
>>>>
>>>> --
>>>> Best, Jingsong Lee
>>>>
>>>

Re: [PROPOSAL] Contribute Stateful Functions to Apache Flink

[Biao Liu <mm...@gmail.com>]

Hi Stehpan,

+1 for having Stateful Functions in Flink.

Before discussing which repository it should belong, I was wondering if we
have reached an agreement of "splitting flink repository" as Piotr
mentioned or not. It seems that it's just no more further discussion.
It's OK for me to add it to core repository. After all almost everything is
in core repository now. But if we decide to split the core repository
someday, I tend to create a separate repository for Stateful Functions. It
might be good time to take the first step of splitting.

Thanks,
Biao /'bɪ.aʊ/



On Sat, 12 Oct 2019 at 19:31, Yu Li <ca...@gmail.com> wrote:

> Hi Stephan,
>
> Big +1 for adding stateful functions to Flink. I believe a lot of user
> would be interested to try this out and I could imagine how this could
> contribute to reduce the TCO for business requiring both streaming
> processing and stateful functions.
>
> And my 2 cents is to put it into flink core repository since I could see a
> tight connection between this library and flink state.
>
> Best Regards,
> Yu
>
>
> On Sat, 12 Oct 2019 at 17:31, jincheng sun <su...@gmail.com>
> wrote:
>
>> Hi Stephan,
>>
>> bit +1 for adding this great features to Apache Flink.
>>
>> Regarding where we should place it, put it into Flink core repository or
>> create a separate repository? I prefer put it into main repository and
>> looking forward the more detail discussion for this decision.
>>
>> Best,
>> Jincheng
>>
>>
>> Jingsong Li <ji...@gmail.com> 于2019年10月12日周六 上午11:32写道：
>>
>>> Hi Stephan,
>>>
>>> big +1 for this contribution. It provides another user interface that is
>>> easy to use and popular at this time. these functions, It's hard for users
>>> to write in SQL/TableApi, while using DataStream is too complex. (We've
>>> done some stateFun kind jobs using DataStream before). With statefun, it is
>>> very easy.
>>>
>>> I think it's also a good opportunity to exercise Flink's core
>>> capabilities. I looked at stateful-functions-flink briefly, it is very
>>> interesting. I think there are many other things Flink can improve. So I
>>> think it's a better thing to put it into Flink, and the improvement for it
>>> will be more natural in the future.
>>>
>>> Best,
>>> Jingsong Lee
>>>
>>> On Fri, Oct 11, 2019 at 7:33 PM Dawid Wysakowicz <dw...@apache.org>
>>> wrote:
>>>
>>>> Hi Stephan,
>>>>
>>>> I think this is a nice library, but what I like more about it is that
>>>> it suggests exploring different use-cases. I think it definitely makes
>>>> sense for the Flink community to explore more lightweight applications that
>>>> reuses resources. Therefore I definitely think it is a good idea for Flink
>>>> community to accept this contribution and help maintaining it.
>>>>
>>>> Personally I'd prefer to have it in a separate repository. There were a
>>>> few discussions before where different people were suggesting to extract
>>>> connectors and other libraries to separate repositories. Moreover I think
>>>> it could serve as an example for the Flink ecosystem website[1]. This could
>>>> be the first project in there and give a good impression that the community
>>>> sees potential in the ecosystem website.
>>>>
>>>> Lastly, I'm wondering if this should go through PMC vote according to
>>>> our bylaws[2]. In the end the suggestion is to adopt an existing code base
>>>> as is. It also proposes a new programs concept that could result in a shift
>>>> of priorities for the community in a long run.
>>>>
>>>> Best,
>>>>
>>>> Dawid
>>>>
>>>> [1]
>>>> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Create-a-Flink-ecosystem-website-td27519.html
>>>>
>>>> [2] https://cwiki.apache.org/confluence/display/FLINK/Flink+Bylaws
>>>> On 11/10/2019 13:12, Till Rohrmann wrote:
>>>>
>>>> Hi Stephan,
>>>>
>>>> +1 for adding stateful functions to Flink. I believe the new set of
>>>> applications this feature will unlock will be super interesting for new and
>>>> existing Flink users alike.
>>>>
>>>> One reason for not including it in the main repository would to not
>>>> being bound to Flink's release cadence. This would allow to release faster
>>>> and more often. However, I believe that having it eventually in Flink's
>>>> main repository would be beneficial in the long run.
>>>>
>>>> Cheers,
>>>> Till
>>>>
>>>> On Fri, Oct 11, 2019 at 12:56 PM Trevor Grant <tr...@gmail.com>
>>>> wrote:
>>>>
>>>>> +1 non-binding on contribution.
>>>>>
>>>>> Separate repo, or feature branch to start maybe? I just feel like in
>>>>> the beginning this thing is going to have lots of breaking changes that
>>>>> maybe aren't going to fit well with tests / other "v1+" release code. Just
>>>>> my .02.
>>>>>
>>>>>
>>>>>
>>>>> On Fri, Oct 11, 2019 at 4:38 AM Stephan Ewen <se...@apache.org> wrote:
>>>>>
>>>>>> Dear Flink Community!
>>>>>>
>>>>>> Some of you probably heard it already: On Tuesday, at Flink Forward
>>>>>> Berlin, we announced **Stateful Functions**.
>>>>>>
>>>>>> Stateful Functions is a library on Flink to implement general purpose
>>>>>> applications. It is built around stateful functions (who would have thunk)
>>>>>> that can communicate arbitrarily through messages, have consistent
>>>>>> state, and a small resource footprint. They are a bit like keyed
>>>>>> ProcessFunctions
>>>>>> that can send each other messages.
>>>>>> As simple as this sounds, this means you can now communicate in
>>>>>> non-DAG patterns, so it allows users to build programs they cannot build
>>>>>> with Flink.
>>>>>> It also has other neat properties, like multiplexing of functions,
>>>>>> modular composition, tooling both container-based deployments and
>>>>>> as-a-Flink-job deployments.
>>>>>>
>>>>>> You can find out more about it here
>>>>>>   - Website: https://statefun.io/
>>>>>>   - Code: https://github.com/ververica/stateful-functions
>>>>>>   - Talk with motivation:
>>>>>> https://speakerdeck.com/stephanewen/stateful-functions-building-general-purpose-applications-and-services-on-apache-flink?slide=12
>>>>>>
>>>>>>
>>>>>> Now for the main issue: **We would like to contribute this project to
>>>>>> Apache Flink**
>>>>>>
>>>>>> I believe that this is a great fit for both sides.
>>>>>> For the Flink community, it would be a way to extend the capabilities
>>>>>> and use cases of Flink into a completely different type of applications and
>>>>>> thus grow the community into this new field.
>>>>>> Many discussions recently about evolving the Flink runtime (both on
>>>>>> the mailing list and at conferences) show the interest in Flink users in
>>>>>> the space that Stateful Functions covers.
>>>>>> It seems natural that Stateful Functions should closely co-develop
>>>>>> with Apache Flink, ideally as part of the project.
>>>>>>
>>>>>> There are many details to be discusses, for example whether this
>>>>>> should be added to the Flink core repository, or whether we and to create a
>>>>>> separate repository
>>>>>> for this. But I think we should start discussing this after we have
>>>>>> consensus on whether the community wants this contribution.
>>>>>>
>>>>>> Really looking forward to hear what you think!
>>>>>>
>>>>>> Best Regards,
>>>>>> Stephan
>>>>>>
>>>>>>
>>>
>>> --
>>> Best, Jingsong Lee
>>>
>>

Re: [PROPOSAL] Contribute Stateful Functions to Apache Flink

[Biao Liu <mm...@gmail.com>]

Hi Stehpan,

+1 for having Stateful Functions in Flink.

Before discussing which repository it should belong, I was wondering if we
have reached an agreement of "splitting flink repository" as Piotr
mentioned or not. It seems that it's just no more further discussion.
It's OK for me to add it to core repository. After all almost everything is
in core repository now. But if we decide to split the core repository
someday, I tend to create a separate repository for Stateful Functions. It
might be good time to take the first step of splitting.

Thanks,
Biao /'bɪ.aʊ/



On Sat, 12 Oct 2019 at 19:31, Yu Li <ca...@gmail.com> wrote:

> Hi Stephan,
>
> Big +1 for adding stateful functions to Flink. I believe a lot of user
> would be interested to try this out and I could imagine how this could
> contribute to reduce the TCO for business requiring both streaming
> processing and stateful functions.
>
> And my 2 cents is to put it into flink core repository since I could see a
> tight connection between this library and flink state.
>
> Best Regards,
> Yu
>
>
> On Sat, 12 Oct 2019 at 17:31, jincheng sun <su...@gmail.com>
> wrote:
>
>> Hi Stephan,
>>
>> bit +1 for adding this great features to Apache Flink.
>>
>> Regarding where we should place it, put it into Flink core repository or
>> create a separate repository? I prefer put it into main repository and
>> looking forward the more detail discussion for this decision.
>>
>> Best,
>> Jincheng
>>
>>
>> Jingsong Li <ji...@gmail.com> 于2019年10月12日周六 上午11:32写道：
>>
>>> Hi Stephan,
>>>
>>> big +1 for this contribution. It provides another user interface that is
>>> easy to use and popular at this time. these functions, It's hard for users
>>> to write in SQL/TableApi, while using DataStream is too complex. (We've
>>> done some stateFun kind jobs using DataStream before). With statefun, it is
>>> very easy.
>>>
>>> I think it's also a good opportunity to exercise Flink's core
>>> capabilities. I looked at stateful-functions-flink briefly, it is very
>>> interesting. I think there are many other things Flink can improve. So I
>>> think it's a better thing to put it into Flink, and the improvement for it
>>> will be more natural in the future.
>>>
>>> Best,
>>> Jingsong Lee
>>>
>>> On Fri, Oct 11, 2019 at 7:33 PM Dawid Wysakowicz <dw...@apache.org>
>>> wrote:
>>>
>>>> Hi Stephan,
>>>>
>>>> I think this is a nice library, but what I like more about it is that
>>>> it suggests exploring different use-cases. I think it definitely makes
>>>> sense for the Flink community to explore more lightweight applications that
>>>> reuses resources. Therefore I definitely think it is a good idea for Flink
>>>> community to accept this contribution and help maintaining it.
>>>>
>>>> Personally I'd prefer to have it in a separate repository. There were a
>>>> few discussions before where different people were suggesting to extract
>>>> connectors and other libraries to separate repositories. Moreover I think
>>>> it could serve as an example for the Flink ecosystem website[1]. This could
>>>> be the first project in there and give a good impression that the community
>>>> sees potential in the ecosystem website.
>>>>
>>>> Lastly, I'm wondering if this should go through PMC vote according to
>>>> our bylaws[2]. In the end the suggestion is to adopt an existing code base
>>>> as is. It also proposes a new programs concept that could result in a shift
>>>> of priorities for the community in a long run.
>>>>
>>>> Best,
>>>>
>>>> Dawid
>>>>
>>>> [1]
>>>> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Create-a-Flink-ecosystem-website-td27519.html
>>>>
>>>> [2] https://cwiki.apache.org/confluence/display/FLINK/Flink+Bylaws
>>>> On 11/10/2019 13:12, Till Rohrmann wrote:
>>>>
>>>> Hi Stephan,
>>>>
>>>> +1 for adding stateful functions to Flink. I believe the new set of
>>>> applications this feature will unlock will be super interesting for new and
>>>> existing Flink users alike.
>>>>
>>>> One reason for not including it in the main repository would to not
>>>> being bound to Flink's release cadence. This would allow to release faster
>>>> and more often. However, I believe that having it eventually in Flink's
>>>> main repository would be beneficial in the long run.
>>>>
>>>> Cheers,
>>>> Till
>>>>
>>>> On Fri, Oct 11, 2019 at 12:56 PM Trevor Grant <tr...@gmail.com>
>>>> wrote:
>>>>
>>>>> +1 non-binding on contribution.
>>>>>
>>>>> Separate repo, or feature branch to start maybe? I just feel like in
>>>>> the beginning this thing is going to have lots of breaking changes that
>>>>> maybe aren't going to fit well with tests / other "v1+" release code. Just
>>>>> my .02.
>>>>>
>>>>>
>>>>>
>>>>> On Fri, Oct 11, 2019 at 4:38 AM Stephan Ewen <se...@apache.org> wrote:
>>>>>
>>>>>> Dear Flink Community!
>>>>>>
>>>>>> Some of you probably heard it already: On Tuesday, at Flink Forward
>>>>>> Berlin, we announced **Stateful Functions**.
>>>>>>
>>>>>> Stateful Functions is a library on Flink to implement general purpose
>>>>>> applications. It is built around stateful functions (who would have thunk)
>>>>>> that can communicate arbitrarily through messages, have consistent
>>>>>> state, and a small resource footprint. They are a bit like keyed
>>>>>> ProcessFunctions
>>>>>> that can send each other messages.
>>>>>> As simple as this sounds, this means you can now communicate in
>>>>>> non-DAG patterns, so it allows users to build programs they cannot build
>>>>>> with Flink.
>>>>>> It also has other neat properties, like multiplexing of functions,
>>>>>> modular composition, tooling both container-based deployments and
>>>>>> as-a-Flink-job deployments.
>>>>>>
>>>>>> You can find out more about it here
>>>>>>   - Website: https://statefun.io/
>>>>>>   - Code: https://github.com/ververica/stateful-functions
>>>>>>   - Talk with motivation:
>>>>>> https://speakerdeck.com/stephanewen/stateful-functions-building-general-purpose-applications-and-services-on-apache-flink?slide=12
>>>>>>
>>>>>>
>>>>>> Now for the main issue: **We would like to contribute this project to
>>>>>> Apache Flink**
>>>>>>
>>>>>> I believe that this is a great fit for both sides.
>>>>>> For the Flink community, it would be a way to extend the capabilities
>>>>>> and use cases of Flink into a completely different type of applications and
>>>>>> thus grow the community into this new field.
>>>>>> Many discussions recently about evolving the Flink runtime (both on
>>>>>> the mailing list and at conferences) show the interest in Flink users in
>>>>>> the space that Stateful Functions covers.
>>>>>> It seems natural that Stateful Functions should closely co-develop
>>>>>> with Apache Flink, ideally as part of the project.
>>>>>>
>>>>>> There are many details to be discusses, for example whether this
>>>>>> should be added to the Flink core repository, or whether we and to create a
>>>>>> separate repository
>>>>>> for this. But I think we should start discussing this after we have
>>>>>> consensus on whether the community wants this contribution.
>>>>>>
>>>>>> Really looking forward to hear what you think!
>>>>>>
>>>>>> Best Regards,
>>>>>> Stephan
>>>>>>
>>>>>>
>>>
>>> --
>>> Best, Jingsong Lee
>>>
>>

Re: [PROPOSAL] Contribute Stateful Functions to Apache Flink

[Yu Li <ca...@gmail.com>]

Hi Stephan,

Big +1 for adding stateful functions to Flink. I believe a lot of user
would be interested to try this out and I could imagine how this could
contribute to reduce the TCO for business requiring both streaming
processing and stateful functions.

And my 2 cents is to put it into flink core repository since I could see a
tight connection between this library and flink state.

Best Regards,
Yu


On Sat, 12 Oct 2019 at 17:31, jincheng sun <su...@gmail.com> wrote:

> Hi Stephan,
>
> bit +1 for adding this great features to Apache Flink.
>
> Regarding where we should place it, put it into Flink core repository or
> create a separate repository? I prefer put it into main repository and
> looking forward the more detail discussion for this decision.
>
> Best,
> Jincheng
>
>
> Jingsong Li <ji...@gmail.com> 于2019年10月12日周六 上午11:32写道：
>
>> Hi Stephan,
>>
>> big +1 for this contribution. It provides another user interface that is
>> easy to use and popular at this time. these functions, It's hard for users
>> to write in SQL/TableApi, while using DataStream is too complex. (We've
>> done some stateFun kind jobs using DataStream before). With statefun, it is
>> very easy.
>>
>> I think it's also a good opportunity to exercise Flink's core
>> capabilities. I looked at stateful-functions-flink briefly, it is very
>> interesting. I think there are many other things Flink can improve. So I
>> think it's a better thing to put it into Flink, and the improvement for it
>> will be more natural in the future.
>>
>> Best,
>> Jingsong Lee
>>
>> On Fri, Oct 11, 2019 at 7:33 PM Dawid Wysakowicz <dw...@apache.org>
>> wrote:
>>
>>> Hi Stephan,
>>>
>>> I think this is a nice library, but what I like more about it is that it
>>> suggests exploring different use-cases. I think it definitely makes sense
>>> for the Flink community to explore more lightweight applications that
>>> reuses resources. Therefore I definitely think it is a good idea for Flink
>>> community to accept this contribution and help maintaining it.
>>>
>>> Personally I'd prefer to have it in a separate repository. There were a
>>> few discussions before where different people were suggesting to extract
>>> connectors and other libraries to separate repositories. Moreover I think
>>> it could serve as an example for the Flink ecosystem website[1]. This could
>>> be the first project in there and give a good impression that the community
>>> sees potential in the ecosystem website.
>>>
>>> Lastly, I'm wondering if this should go through PMC vote according to
>>> our bylaws[2]. In the end the suggestion is to adopt an existing code base
>>> as is. It also proposes a new programs concept that could result in a shift
>>> of priorities for the community in a long run.
>>>
>>> Best,
>>>
>>> Dawid
>>>
>>> [1]
>>> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Create-a-Flink-ecosystem-website-td27519.html
>>>
>>> [2] https://cwiki.apache.org/confluence/display/FLINK/Flink+Bylaws
>>> On 11/10/2019 13:12, Till Rohrmann wrote:
>>>
>>> Hi Stephan,
>>>
>>> +1 for adding stateful functions to Flink. I believe the new set of
>>> applications this feature will unlock will be super interesting for new and
>>> existing Flink users alike.
>>>
>>> One reason for not including it in the main repository would to not
>>> being bound to Flink's release cadence. This would allow to release faster
>>> and more often. However, I believe that having it eventually in Flink's
>>> main repository would be beneficial in the long run.
>>>
>>> Cheers,
>>> Till
>>>
>>> On Fri, Oct 11, 2019 at 12:56 PM Trevor Grant <tr...@gmail.com>
>>> wrote:
>>>
>>>> +1 non-binding on contribution.
>>>>
>>>> Separate repo, or feature branch to start maybe? I just feel like in
>>>> the beginning this thing is going to have lots of breaking changes that
>>>> maybe aren't going to fit well with tests / other "v1+" release code. Just
>>>> my .02.
>>>>
>>>>
>>>>
>>>> On Fri, Oct 11, 2019 at 4:38 AM Stephan Ewen <se...@apache.org> wrote:
>>>>
>>>>> Dear Flink Community!
>>>>>
>>>>> Some of you probably heard it already: On Tuesday, at Flink Forward
>>>>> Berlin, we announced **Stateful Functions**.
>>>>>
>>>>> Stateful Functions is a library on Flink to implement general purpose
>>>>> applications. It is built around stateful functions (who would have thunk)
>>>>> that can communicate arbitrarily through messages, have consistent
>>>>> state, and a small resource footprint. They are a bit like keyed
>>>>> ProcessFunctions
>>>>> that can send each other messages.
>>>>> As simple as this sounds, this means you can now communicate in
>>>>> non-DAG patterns, so it allows users to build programs they cannot build
>>>>> with Flink.
>>>>> It also has other neat properties, like multiplexing of functions,
>>>>> modular composition, tooling both container-based deployments and
>>>>> as-a-Flink-job deployments.
>>>>>
>>>>> You can find out more about it here
>>>>>   - Website: https://statefun.io/
>>>>>   - Code: https://github.com/ververica/stateful-functions
>>>>>   - Talk with motivation:
>>>>> https://speakerdeck.com/stephanewen/stateful-functions-building-general-purpose-applications-and-services-on-apache-flink?slide=12
>>>>>
>>>>>
>>>>> Now for the main issue: **We would like to contribute this project to
>>>>> Apache Flink**
>>>>>
>>>>> I believe that this is a great fit for both sides.
>>>>> For the Flink community, it would be a way to extend the capabilities
>>>>> and use cases of Flink into a completely different type of applications and
>>>>> thus grow the community into this new field.
>>>>> Many discussions recently about evolving the Flink runtime (both on
>>>>> the mailing list and at conferences) show the interest in Flink users in
>>>>> the space that Stateful Functions covers.
>>>>> It seems natural that Stateful Functions should closely co-develop
>>>>> with Apache Flink, ideally as part of the project.
>>>>>
>>>>> There are many details to be discusses, for example whether this
>>>>> should be added to the Flink core repository, or whether we and to create a
>>>>> separate repository
>>>>> for this. But I think we should start discussing this after we have
>>>>> consensus on whether the community wants this contribution.
>>>>>
>>>>> Really looking forward to hear what you think!
>>>>>
>>>>> Best Regards,
>>>>> Stephan
>>>>>
>>>>>
>>
>> --
>> Best, Jingsong Lee
>>
>

Re: [PROPOSAL] Contribute Stateful Functions to Apache Flink

[Yu Li <ca...@gmail.com>]

Hi Stephan,

Big +1 for adding stateful functions to Flink. I believe a lot of user
would be interested to try this out and I could imagine how this could
contribute to reduce the TCO for business requiring both streaming
processing and stateful functions.

And my 2 cents is to put it into flink core repository since I could see a
tight connection between this library and flink state.

Best Regards,
Yu


On Sat, 12 Oct 2019 at 17:31, jincheng sun <su...@gmail.com> wrote:

> Hi Stephan,
>
> bit +1 for adding this great features to Apache Flink.
>
> Regarding where we should place it, put it into Flink core repository or
> create a separate repository? I prefer put it into main repository and
> looking forward the more detail discussion for this decision.
>
> Best,
> Jincheng
>
>
> Jingsong Li <ji...@gmail.com> 于2019年10月12日周六 上午11:32写道：
>
>> Hi Stephan,
>>
>> big +1 for this contribution. It provides another user interface that is
>> easy to use and popular at this time. these functions, It's hard for users
>> to write in SQL/TableApi, while using DataStream is too complex. (We've
>> done some stateFun kind jobs using DataStream before). With statefun, it is
>> very easy.
>>
>> I think it's also a good opportunity to exercise Flink's core
>> capabilities. I looked at stateful-functions-flink briefly, it is very
>> interesting. I think there are many other things Flink can improve. So I
>> think it's a better thing to put it into Flink, and the improvement for it
>> will be more natural in the future.
>>
>> Best,
>> Jingsong Lee
>>
>> On Fri, Oct 11, 2019 at 7:33 PM Dawid Wysakowicz <dw...@apache.org>
>> wrote:
>>
>>> Hi Stephan,
>>>
>>> I think this is a nice library, but what I like more about it is that it
>>> suggests exploring different use-cases. I think it definitely makes sense
>>> for the Flink community to explore more lightweight applications that
>>> reuses resources. Therefore I definitely think it is a good idea for Flink
>>> community to accept this contribution and help maintaining it.
>>>
>>> Personally I'd prefer to have it in a separate repository. There were a
>>> few discussions before where different people were suggesting to extract
>>> connectors and other libraries to separate repositories. Moreover I think
>>> it could serve as an example for the Flink ecosystem website[1]. This could
>>> be the first project in there and give a good impression that the community
>>> sees potential in the ecosystem website.
>>>
>>> Lastly, I'm wondering if this should go through PMC vote according to
>>> our bylaws[2]. In the end the suggestion is to adopt an existing code base
>>> as is. It also proposes a new programs concept that could result in a shift
>>> of priorities for the community in a long run.
>>>
>>> Best,
>>>
>>> Dawid
>>>
>>> [1]
>>> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Create-a-Flink-ecosystem-website-td27519.html
>>>
>>> [2] https://cwiki.apache.org/confluence/display/FLINK/Flink+Bylaws
>>> On 11/10/2019 13:12, Till Rohrmann wrote:
>>>
>>> Hi Stephan,
>>>
>>> +1 for adding stateful functions to Flink. I believe the new set of
>>> applications this feature will unlock will be super interesting for new and
>>> existing Flink users alike.
>>>
>>> One reason for not including it in the main repository would to not
>>> being bound to Flink's release cadence. This would allow to release faster
>>> and more often. However, I believe that having it eventually in Flink's
>>> main repository would be beneficial in the long run.
>>>
>>> Cheers,
>>> Till
>>>
>>> On Fri, Oct 11, 2019 at 12:56 PM Trevor Grant <tr...@gmail.com>
>>> wrote:
>>>
>>>> +1 non-binding on contribution.
>>>>
>>>> Separate repo, or feature branch to start maybe? I just feel like in
>>>> the beginning this thing is going to have lots of breaking changes that
>>>> maybe aren't going to fit well with tests / other "v1+" release code. Just
>>>> my .02.
>>>>
>>>>
>>>>
>>>> On Fri, Oct 11, 2019 at 4:38 AM Stephan Ewen <se...@apache.org> wrote:
>>>>
>>>>> Dear Flink Community!
>>>>>
>>>>> Some of you probably heard it already: On Tuesday, at Flink Forward
>>>>> Berlin, we announced **Stateful Functions**.
>>>>>
>>>>> Stateful Functions is a library on Flink to implement general purpose
>>>>> applications. It is built around stateful functions (who would have thunk)
>>>>> that can communicate arbitrarily through messages, have consistent
>>>>> state, and a small resource footprint. They are a bit like keyed
>>>>> ProcessFunctions
>>>>> that can send each other messages.
>>>>> As simple as this sounds, this means you can now communicate in
>>>>> non-DAG patterns, so it allows users to build programs they cannot build
>>>>> with Flink.
>>>>> It also has other neat properties, like multiplexing of functions,
>>>>> modular composition, tooling both container-based deployments and
>>>>> as-a-Flink-job deployments.
>>>>>
>>>>> You can find out more about it here
>>>>>   - Website: https://statefun.io/
>>>>>   - Code: https://github.com/ververica/stateful-functions
>>>>>   - Talk with motivation:
>>>>> https://speakerdeck.com/stephanewen/stateful-functions-building-general-purpose-applications-and-services-on-apache-flink?slide=12
>>>>>
>>>>>
>>>>> Now for the main issue: **We would like to contribute this project to
>>>>> Apache Flink**
>>>>>
>>>>> I believe that this is a great fit for both sides.
>>>>> For the Flink community, it would be a way to extend the capabilities
>>>>> and use cases of Flink into a completely different type of applications and
>>>>> thus grow the community into this new field.
>>>>> Many discussions recently about evolving the Flink runtime (both on
>>>>> the mailing list and at conferences) show the interest in Flink users in
>>>>> the space that Stateful Functions covers.
>>>>> It seems natural that Stateful Functions should closely co-develop
>>>>> with Apache Flink, ideally as part of the project.
>>>>>
>>>>> There are many details to be discusses, for example whether this
>>>>> should be added to the Flink core repository, or whether we and to create a
>>>>> separate repository
>>>>> for this. But I think we should start discussing this after we have
>>>>> consensus on whether the community wants this contribution.
>>>>>
>>>>> Really looking forward to hear what you think!
>>>>>
>>>>> Best Regards,
>>>>> Stephan
>>>>>
>>>>>
>>
>> --
>> Best, Jingsong Lee
>>
>

Re: [PROPOSAL] Contribute Stateful Functions to Apache Flink

[jincheng sun <su...@gmail.com>]

Hi Stephan,

bit +1 for adding this great features to Apache Flink.

Regarding where we should place it, put it into Flink core repository or
create a separate repository? I prefer put it into main repository and
looking forward the more detail discussion for this decision.

Best,
Jincheng


Jingsong Li <ji...@gmail.com> 于2019年10月12日周六 上午11:32写道：

> Hi Stephan,
>
> big +1 for this contribution. It provides another user interface that is
> easy to use and popular at this time. these functions, It's hard for users
> to write in SQL/TableApi, while using DataStream is too complex. (We've
> done some stateFun kind jobs using DataStream before). With statefun, it is
> very easy.
>
> I think it's also a good opportunity to exercise Flink's core
> capabilities. I looked at stateful-functions-flink briefly, it is very
> interesting. I think there are many other things Flink can improve. So I
> think it's a better thing to put it into Flink, and the improvement for it
> will be more natural in the future.
>
> Best,
> Jingsong Lee
>
> On Fri, Oct 11, 2019 at 7:33 PM Dawid Wysakowicz <dw...@apache.org>
> wrote:
>
>> Hi Stephan,
>>
>> I think this is a nice library, but what I like more about it is that it
>> suggests exploring different use-cases. I think it definitely makes sense
>> for the Flink community to explore more lightweight applications that
>> reuses resources. Therefore I definitely think it is a good idea for Flink
>> community to accept this contribution and help maintaining it.
>>
>> Personally I'd prefer to have it in a separate repository. There were a
>> few discussions before where different people were suggesting to extract
>> connectors and other libraries to separate repositories. Moreover I think
>> it could serve as an example for the Flink ecosystem website[1]. This could
>> be the first project in there and give a good impression that the community
>> sees potential in the ecosystem website.
>>
>> Lastly, I'm wondering if this should go through PMC vote according to our
>> bylaws[2]. In the end the suggestion is to adopt an existing code base as
>> is. It also proposes a new programs concept that could result in a shift of
>> priorities for the community in a long run.
>>
>> Best,
>>
>> Dawid
>>
>> [1]
>> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Create-a-Flink-ecosystem-website-td27519.html
>>
>> [2] https://cwiki.apache.org/confluence/display/FLINK/Flink+Bylaws
>> On 11/10/2019 13:12, Till Rohrmann wrote:
>>
>> Hi Stephan,
>>
>> +1 for adding stateful functions to Flink. I believe the new set of
>> applications this feature will unlock will be super interesting for new and
>> existing Flink users alike.
>>
>> One reason for not including it in the main repository would to not being
>> bound to Flink's release cadence. This would allow to release faster and
>> more often. However, I believe that having it eventually in Flink's main
>> repository would be beneficial in the long run.
>>
>> Cheers,
>> Till
>>
>> On Fri, Oct 11, 2019 at 12:56 PM Trevor Grant <tr...@gmail.com>
>> wrote:
>>
>>> +1 non-binding on contribution.
>>>
>>> Separate repo, or feature branch to start maybe? I just feel like in the
>>> beginning this thing is going to have lots of breaking changes that maybe
>>> aren't going to fit well with tests / other "v1+" release code. Just my
>>> .02.
>>>
>>>
>>>
>>> On Fri, Oct 11, 2019 at 4:38 AM Stephan Ewen <se...@apache.org> wrote:
>>>
>>>> Dear Flink Community!
>>>>
>>>> Some of you probably heard it already: On Tuesday, at Flink Forward
>>>> Berlin, we announced **Stateful Functions**.
>>>>
>>>> Stateful Functions is a library on Flink to implement general purpose
>>>> applications. It is built around stateful functions (who would have thunk)
>>>> that can communicate arbitrarily through messages, have consistent
>>>> state, and a small resource footprint. They are a bit like keyed
>>>> ProcessFunctions
>>>> that can send each other messages.
>>>> As simple as this sounds, this means you can now communicate in non-DAG
>>>> patterns, so it allows users to build programs they cannot build with Flink.
>>>> It also has other neat properties, like multiplexing of functions,
>>>> modular composition, tooling both container-based deployments and
>>>> as-a-Flink-job deployments.
>>>>
>>>> You can find out more about it here
>>>>   - Website: https://statefun.io/
>>>>   - Code: https://github.com/ververica/stateful-functions
>>>>   - Talk with motivation:
>>>> https://speakerdeck.com/stephanewen/stateful-functions-building-general-purpose-applications-and-services-on-apache-flink?slide=12
>>>>
>>>>
>>>> Now for the main issue: **We would like to contribute this project to
>>>> Apache Flink**
>>>>
>>>> I believe that this is a great fit for both sides.
>>>> For the Flink community, it would be a way to extend the capabilities
>>>> and use cases of Flink into a completely different type of applications and
>>>> thus grow the community into this new field.
>>>> Many discussions recently about evolving the Flink runtime (both on the
>>>> mailing list and at conferences) show the interest in Flink users in the
>>>> space that Stateful Functions covers.
>>>> It seems natural that Stateful Functions should closely co-develop with
>>>> Apache Flink, ideally as part of the project.
>>>>
>>>> There are many details to be discusses, for example whether this should
>>>> be added to the Flink core repository, or whether we and to create a
>>>> separate repository
>>>> for this. But I think we should start discussing this after we have
>>>> consensus on whether the community wants this contribution.
>>>>
>>>> Really looking forward to hear what you think!
>>>>
>>>> Best Regards,
>>>> Stephan
>>>>
>>>>
>
> --
> Best, Jingsong Lee
>

Re: [PROPOSAL] Contribute Stateful Functions to Apache Flink

[jincheng sun <su...@gmail.com>]

Hi Stephan,

bit +1 for adding this great features to Apache Flink.

Regarding where we should place it, put it into Flink core repository or
create a separate repository? I prefer put it into main repository and
looking forward the more detail discussion for this decision.

Best,
Jincheng


Jingsong Li <ji...@gmail.com> 于2019年10月12日周六 上午11:32写道：

> Hi Stephan,
>
> big +1 for this contribution. It provides another user interface that is
> easy to use and popular at this time. these functions, It's hard for users
> to write in SQL/TableApi, while using DataStream is too complex. (We've
> done some stateFun kind jobs using DataStream before). With statefun, it is
> very easy.
>
> I think it's also a good opportunity to exercise Flink's core
> capabilities. I looked at stateful-functions-flink briefly, it is very
> interesting. I think there are many other things Flink can improve. So I
> think it's a better thing to put it into Flink, and the improvement for it
> will be more natural in the future.
>
> Best,
> Jingsong Lee
>
> On Fri, Oct 11, 2019 at 7:33 PM Dawid Wysakowicz <dw...@apache.org>
> wrote:
>
>> Hi Stephan,
>>
>> I think this is a nice library, but what I like more about it is that it
>> suggests exploring different use-cases. I think it definitely makes sense
>> for the Flink community to explore more lightweight applications that
>> reuses resources. Therefore I definitely think it is a good idea for Flink
>> community to accept this contribution and help maintaining it.
>>
>> Personally I'd prefer to have it in a separate repository. There were a
>> few discussions before where different people were suggesting to extract
>> connectors and other libraries to separate repositories. Moreover I think
>> it could serve as an example for the Flink ecosystem website[1]. This could
>> be the first project in there and give a good impression that the community
>> sees potential in the ecosystem website.
>>
>> Lastly, I'm wondering if this should go through PMC vote according to our
>> bylaws[2]. In the end the suggestion is to adopt an existing code base as
>> is. It also proposes a new programs concept that could result in a shift of
>> priorities for the community in a long run.
>>
>> Best,
>>
>> Dawid
>>
>> [1]
>> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Create-a-Flink-ecosystem-website-td27519.html
>>
>> [2] https://cwiki.apache.org/confluence/display/FLINK/Flink+Bylaws
>> On 11/10/2019 13:12, Till Rohrmann wrote:
>>
>> Hi Stephan,
>>
>> +1 for adding stateful functions to Flink. I believe the new set of
>> applications this feature will unlock will be super interesting for new and
>> existing Flink users alike.
>>
>> One reason for not including it in the main repository would to not being
>> bound to Flink's release cadence. This would allow to release faster and
>> more often. However, I believe that having it eventually in Flink's main
>> repository would be beneficial in the long run.
>>
>> Cheers,
>> Till
>>
>> On Fri, Oct 11, 2019 at 12:56 PM Trevor Grant <tr...@gmail.com>
>> wrote:
>>
>>> +1 non-binding on contribution.
>>>
>>> Separate repo, or feature branch to start maybe? I just feel like in the
>>> beginning this thing is going to have lots of breaking changes that maybe
>>> aren't going to fit well with tests / other "v1+" release code. Just my
>>> .02.
>>>
>>>
>>>
>>> On Fri, Oct 11, 2019 at 4:38 AM Stephan Ewen <se...@apache.org> wrote:
>>>
>>>> Dear Flink Community!
>>>>
>>>> Some of you probably heard it already: On Tuesday, at Flink Forward
>>>> Berlin, we announced **Stateful Functions**.
>>>>
>>>> Stateful Functions is a library on Flink to implement general purpose
>>>> applications. It is built around stateful functions (who would have thunk)
>>>> that can communicate arbitrarily through messages, have consistent
>>>> state, and a small resource footprint. They are a bit like keyed
>>>> ProcessFunctions
>>>> that can send each other messages.
>>>> As simple as this sounds, this means you can now communicate in non-DAG
>>>> patterns, so it allows users to build programs they cannot build with Flink.
>>>> It also has other neat properties, like multiplexing of functions,
>>>> modular composition, tooling both container-based deployments and
>>>> as-a-Flink-job deployments.
>>>>
>>>> You can find out more about it here
>>>>   - Website: https://statefun.io/
>>>>   - Code: https://github.com/ververica/stateful-functions
>>>>   - Talk with motivation:
>>>> https://speakerdeck.com/stephanewen/stateful-functions-building-general-purpose-applications-and-services-on-apache-flink?slide=12
>>>>
>>>>
>>>> Now for the main issue: **We would like to contribute this project to
>>>> Apache Flink**
>>>>
>>>> I believe that this is a great fit for both sides.
>>>> For the Flink community, it would be a way to extend the capabilities
>>>> and use cases of Flink into a completely different type of applications and
>>>> thus grow the community into this new field.
>>>> Many discussions recently about evolving the Flink runtime (both on the
>>>> mailing list and at conferences) show the interest in Flink users in the
>>>> space that Stateful Functions covers.
>>>> It seems natural that Stateful Functions should closely co-develop with
>>>> Apache Flink, ideally as part of the project.
>>>>
>>>> There are many details to be discusses, for example whether this should
>>>> be added to the Flink core repository, or whether we and to create a
>>>> separate repository
>>>> for this. But I think we should start discussing this after we have
>>>> consensus on whether the community wants this contribution.
>>>>
>>>> Really looking forward to hear what you think!
>>>>
>>>> Best Regards,
>>>> Stephan
>>>>
>>>>
>
> --
> Best, Jingsong Lee
>

Re: [PROPOSAL] Contribute Stateful Functions to Apache Flink

[Jingsong Li <ji...@gmail.com>]

Hi Stephan,

big +1 for this contribution. It provides another user interface that is
easy to use and popular at this time. these functions, It's hard for users
to write in SQL/TableApi, while using DataStream is too complex. (We've
done some stateFun kind jobs using DataStream before). With statefun, it is
very easy.

I think it's also a good opportunity to exercise Flink's core capabilities.
I looked at stateful-functions-flink briefly, it is very interesting. I
think there are many other things Flink can improve. So I think it's a
better thing to put it into Flink, and the improvement for it will be more
natural in the future.

Best,
Jingsong Lee

On Fri, Oct 11, 2019 at 7:33 PM Dawid Wysakowicz <dw...@apache.org>
wrote:

> Hi Stephan,
>
> I think this is a nice library, but what I like more about it is that it
> suggests exploring different use-cases. I think it definitely makes sense
> for the Flink community to explore more lightweight applications that
> reuses resources. Therefore I definitely think it is a good idea for Flink
> community to accept this contribution and help maintaining it.
>
> Personally I'd prefer to have it in a separate repository. There were a
> few discussions before where different people were suggesting to extract
> connectors and other libraries to separate repositories. Moreover I think
> it could serve as an example for the Flink ecosystem website[1]. This could
> be the first project in there and give a good impression that the community
> sees potential in the ecosystem website.
>
> Lastly, I'm wondering if this should go through PMC vote according to our
> bylaws[2]. In the end the suggestion is to adopt an existing code base as
> is. It also proposes a new programs concept that could result in a shift of
> priorities for the community in a long run.
>
> Best,
>
> Dawid
>
> [1]
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Create-a-Flink-ecosystem-website-td27519.html
>
> [2] https://cwiki.apache.org/confluence/display/FLINK/Flink+Bylaws
> On 11/10/2019 13:12, Till Rohrmann wrote:
>
> Hi Stephan,
>
> +1 for adding stateful functions to Flink. I believe the new set of
> applications this feature will unlock will be super interesting for new and
> existing Flink users alike.
>
> One reason for not including it in the main repository would to not being
> bound to Flink's release cadence. This would allow to release faster and
> more often. However, I believe that having it eventually in Flink's main
> repository would be beneficial in the long run.
>
> Cheers,
> Till
>
> On Fri, Oct 11, 2019 at 12:56 PM Trevor Grant <tr...@gmail.com>
> wrote:
>
>> +1 non-binding on contribution.
>>
>> Separate repo, or feature branch to start maybe? I just feel like in the
>> beginning this thing is going to have lots of breaking changes that maybe
>> aren't going to fit well with tests / other "v1+" release code. Just my
>> .02.
>>
>>
>>
>> On Fri, Oct 11, 2019 at 4:38 AM Stephan Ewen <se...@apache.org> wrote:
>>
>>> Dear Flink Community!
>>>
>>> Some of you probably heard it already: On Tuesday, at Flink Forward
>>> Berlin, we announced **Stateful Functions**.
>>>
>>> Stateful Functions is a library on Flink to implement general purpose
>>> applications. It is built around stateful functions (who would have thunk)
>>> that can communicate arbitrarily through messages, have consistent
>>> state, and a small resource footprint. They are a bit like keyed
>>> ProcessFunctions
>>> that can send each other messages.
>>> As simple as this sounds, this means you can now communicate in non-DAG
>>> patterns, so it allows users to build programs they cannot build with Flink.
>>> It also has other neat properties, like multiplexing of functions,
>>> modular composition, tooling both container-based deployments and
>>> as-a-Flink-job deployments.
>>>
>>> You can find out more about it here
>>>   - Website: https://statefun.io/
>>>   - Code: https://github.com/ververica/stateful-functions
>>>   - Talk with motivation:
>>> https://speakerdeck.com/stephanewen/stateful-functions-building-general-purpose-applications-and-services-on-apache-flink?slide=12
>>>
>>>
>>> Now for the main issue: **We would like to contribute this project to
>>> Apache Flink**
>>>
>>> I believe that this is a great fit for both sides.
>>> For the Flink community, it would be a way to extend the capabilities
>>> and use cases of Flink into a completely different type of applications and
>>> thus grow the community into this new field.
>>> Many discussions recently about evolving the Flink runtime (both on the
>>> mailing list and at conferences) show the interest in Flink users in the
>>> space that Stateful Functions covers.
>>> It seems natural that Stateful Functions should closely co-develop with
>>> Apache Flink, ideally as part of the project.
>>>
>>> There are many details to be discusses, for example whether this should
>>> be added to the Flink core repository, or whether we and to create a
>>> separate repository
>>> for this. But I think we should start discussing this after we have
>>> consensus on whether the community wants this contribution.
>>>
>>> Really looking forward to hear what you think!
>>>
>>> Best Regards,
>>> Stephan
>>>
>>>

-- 
Best, Jingsong Lee

Re: [PROPOSAL] Contribute Stateful Functions to Apache Flink

[Jingsong Li <ji...@gmail.com>]

Hi Stephan,

big +1 for this contribution. It provides another user interface that is
easy to use and popular at this time. these functions, It's hard for users
to write in SQL/TableApi, while using DataStream is too complex. (We've
done some stateFun kind jobs using DataStream before). With statefun, it is
very easy.

I think it's also a good opportunity to exercise Flink's core capabilities.
I looked at stateful-functions-flink briefly, it is very interesting. I
think there are many other things Flink can improve. So I think it's a
better thing to put it into Flink, and the improvement for it will be more
natural in the future.

Best,
Jingsong Lee

On Fri, Oct 11, 2019 at 7:33 PM Dawid Wysakowicz <dw...@apache.org>
wrote:

> Hi Stephan,
>
> I think this is a nice library, but what I like more about it is that it
> suggests exploring different use-cases. I think it definitely makes sense
> for the Flink community to explore more lightweight applications that
> reuses resources. Therefore I definitely think it is a good idea for Flink
> community to accept this contribution and help maintaining it.
>
> Personally I'd prefer to have it in a separate repository. There were a
> few discussions before where different people were suggesting to extract
> connectors and other libraries to separate repositories. Moreover I think
> it could serve as an example for the Flink ecosystem website[1]. This could
> be the first project in there and give a good impression that the community
> sees potential in the ecosystem website.
>
> Lastly, I'm wondering if this should go through PMC vote according to our
> bylaws[2]. In the end the suggestion is to adopt an existing code base as
> is. It also proposes a new programs concept that could result in a shift of
> priorities for the community in a long run.
>
> Best,
>
> Dawid
>
> [1]
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Create-a-Flink-ecosystem-website-td27519.html
>
> [2] https://cwiki.apache.org/confluence/display/FLINK/Flink+Bylaws
> On 11/10/2019 13:12, Till Rohrmann wrote:
>
> Hi Stephan,
>
> +1 for adding stateful functions to Flink. I believe the new set of
> applications this feature will unlock will be super interesting for new and
> existing Flink users alike.
>
> One reason for not including it in the main repository would to not being
> bound to Flink's release cadence. This would allow to release faster and
> more often. However, I believe that having it eventually in Flink's main
> repository would be beneficial in the long run.
>
> Cheers,
> Till
>
> On Fri, Oct 11, 2019 at 12:56 PM Trevor Grant <tr...@gmail.com>
> wrote:
>
>> +1 non-binding on contribution.
>>
>> Separate repo, or feature branch to start maybe? I just feel like in the
>> beginning this thing is going to have lots of breaking changes that maybe
>> aren't going to fit well with tests / other "v1+" release code. Just my
>> .02.
>>
>>
>>
>> On Fri, Oct 11, 2019 at 4:38 AM Stephan Ewen <se...@apache.org> wrote:
>>
>>> Dear Flink Community!
>>>
>>> Some of you probably heard it already: On Tuesday, at Flink Forward
>>> Berlin, we announced **Stateful Functions**.
>>>
>>> Stateful Functions is a library on Flink to implement general purpose
>>> applications. It is built around stateful functions (who would have thunk)
>>> that can communicate arbitrarily through messages, have consistent
>>> state, and a small resource footprint. They are a bit like keyed
>>> ProcessFunctions
>>> that can send each other messages.
>>> As simple as this sounds, this means you can now communicate in non-DAG
>>> patterns, so it allows users to build programs they cannot build with Flink.
>>> It also has other neat properties, like multiplexing of functions,
>>> modular composition, tooling both container-based deployments and
>>> as-a-Flink-job deployments.
>>>
>>> You can find out more about it here
>>>   - Website: https://statefun.io/
>>>   - Code: https://github.com/ververica/stateful-functions
>>>   - Talk with motivation:
>>> https://speakerdeck.com/stephanewen/stateful-functions-building-general-purpose-applications-and-services-on-apache-flink?slide=12
>>>
>>>
>>> Now for the main issue: **We would like to contribute this project to
>>> Apache Flink**
>>>
>>> I believe that this is a great fit for both sides.
>>> For the Flink community, it would be a way to extend the capabilities
>>> and use cases of Flink into a completely different type of applications and
>>> thus grow the community into this new field.
>>> Many discussions recently about evolving the Flink runtime (both on the
>>> mailing list and at conferences) show the interest in Flink users in the
>>> space that Stateful Functions covers.
>>> It seems natural that Stateful Functions should closely co-develop with
>>> Apache Flink, ideally as part of the project.
>>>
>>> There are many details to be discusses, for example whether this should
>>> be added to the Flink core repository, or whether we and to create a
>>> separate repository
>>> for this. But I think we should start discussing this after we have
>>> consensus on whether the community wants this contribution.
>>>
>>> Really looking forward to hear what you think!
>>>
>>> Best Regards,
>>> Stephan
>>>
>>>

-- 
Best, Jingsong Lee

Re: [PROPOSAL] Contribute Stateful Functions to Apache Flink

[Dawid Wysakowicz <dw...@apache.org>]

Hi Stephan,

I think this is a nice library, but what I like more about it is that it
suggests exploring different use-cases. I think it definitely makes
sense for the Flink community to explore more lightweight applications
that reuses resources. Therefore I definitely think it is a good idea
for Flink community to accept this contribution and help maintaining it.

Personally I'd prefer to have it in a separate repository. There were a
few discussions before where different people were suggesting to extract
connectors and other libraries to separate repositories. Moreover I
think it could serve as an example for the Flink ecosystem website[1].
This could be the first project in there and give a good impression that
the community sees potential in the ecosystem website.

Lastly, I'm wondering if this should go through PMC vote according to
our bylaws[2]. In the end the suggestion is to adopt an existing code
base as is. It also proposes a new programs concept that could result in
a shift of priorities for the community in a long run.

Best,

Dawid

[1]
http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Create-a-Flink-ecosystem-website-td27519.html

[2] https://cwiki.apache.org/confluence/display/FLINK/Flink+Bylaws

On 11/10/2019 13:12, Till Rohrmann wrote:
> Hi Stephan,
>
> +1 for adding stateful functions to Flink. I believe the new set of
> applications this feature will unlock will be super interesting for
> new and existing Flink users alike.
>
> One reason for not including it in the main repository would to not
> being bound to Flink's release cadence. This would allow to release
> faster and more often. However, I believe that having it eventually in
> Flink's main repository would be beneficial in the long run.
>
> Cheers,
> Till
>
> On Fri, Oct 11, 2019 at 12:56 PM Trevor Grant
> <trevor.d.grant@gmail.com <ma...@gmail.com>> wrote:
>
>     +1 non-binding on contribution. 
>
>     Separate repo, or feature branch to start maybe? I just feel like
>     in the beginning this thing is going to have lots of breaking
>     changes that maybe aren't going to fit well with tests /
>     other "v1+" release code. Just my .02. 
>
>
>
>     On Fri, Oct 11, 2019 at 4:38 AM Stephan Ewen <sewen@apache.org
>     <ma...@apache.org>> wrote:
>
>         Dear Flink Community!
>
>         Some of you probably heard it already: On Tuesday, at Flink
>         Forward Berlin, we announced **Stateful Functions**.
>
>         Stateful Functions is a library on Flink to implement general
>         purpose applications. It is built around stateful functions
>         (who would have thunk)
>         that can communicate arbitrarily through messages, have
>         consistent state, and a small resource footprint. They are a
>         bit like keyed ProcessFunctions
>         that can send each other messages.
>         As simple as this sounds, this means you can now communicate
>         in non-DAG patterns, so it allows users to build programs they
>         cannot build with Flink.
>         It also has other neat properties, like multiplexing of
>         functions, modular composition, tooling both container-based
>         deployments and as-a-Flink-job deployments.
>
>         You can find out more about it here
>           - Website: https://statefun.io/
>           - Code: https://github.com/ververica/stateful-functions
>           - Talk with
>         motivation: https://speakerdeck.com/stephanewen/stateful-functions-building-general-purpose-applications-and-services-on-apache-flink?slide=12
>
>
>         Now for the main issue: **We would like to contribute this
>         project to Apache Flink**
>
>         I believe that this is a great fit for both sides.
>         For the Flink community, it would be a way to extend the
>         capabilities and use cases of Flink into a completely
>         different type of applications and thus grow the community
>         into this new field.
>         Many discussions recently about evolving the Flink runtime
>         (both on the mailing list and at conferences) show the
>         interest in Flink users in the space that Stateful Functions
>         covers.
>         It seems natural that Stateful Functions should closely
>         co-develop with Apache Flink, ideally as part of the project.
>
>         There are many details to be discusses, for example whether
>         this should be added to the Flink core repository, or whether
>         we and to create a separate repository
>         for this. But I think we should start discussing this after we
>         have consensus on whether the community wants this contribution.
>
>         Really looking forward to hear what you think!
>
>         Best Regards,
>         Stephan
>

Re: [PROPOSAL] Contribute Stateful Functions to Apache Flink

[Till Rohrmann <tr...@apache.org>]

Hi Stephan,

+1 for adding stateful functions to Flink. I believe the new set of
applications this feature will unlock will be super interesting for new and
existing Flink users alike.

One reason for not including it in the main repository would to not being
bound to Flink's release cadence. This would allow to release faster and
more often. However, I believe that having it eventually in Flink's main
repository would be beneficial in the long run.

Cheers,
Till

On Fri, Oct 11, 2019 at 12:56 PM Trevor Grant <tr...@gmail.com>
wrote:

> +1 non-binding on contribution.
>
> Separate repo, or feature branch to start maybe? I just feel like in the
> beginning this thing is going to have lots of breaking changes that maybe
> aren't going to fit well with tests / other "v1+" release code. Just my
> .02.
>
>
>
> On Fri, Oct 11, 2019 at 4:38 AM Stephan Ewen <se...@apache.org> wrote:
>
>> Dear Flink Community!
>>
>> Some of you probably heard it already: On Tuesday, at Flink Forward
>> Berlin, we announced **Stateful Functions**.
>>
>> Stateful Functions is a library on Flink to implement general purpose
>> applications. It is built around stateful functions (who would have thunk)
>> that can communicate arbitrarily through messages, have consistent state,
>> and a small resource footprint. They are a bit like keyed ProcessFunctions
>> that can send each other messages.
>> As simple as this sounds, this means you can now communicate in non-DAG
>> patterns, so it allows users to build programs they cannot build with Flink.
>> It also has other neat properties, like multiplexing of functions,
>> modular composition, tooling both container-based deployments and
>> as-a-Flink-job deployments.
>>
>> You can find out more about it here
>>   - Website: https://statefun.io/
>>   - Code: https://github.com/ververica/stateful-functions
>>   - Talk with motivation:
>> https://speakerdeck.com/stephanewen/stateful-functions-building-general-purpose-applications-and-services-on-apache-flink?slide=12
>>
>>
>> Now for the main issue: **We would like to contribute this project to
>> Apache Flink**
>>
>> I believe that this is a great fit for both sides.
>> For the Flink community, it would be a way to extend the capabilities and
>> use cases of Flink into a completely different type of applications and
>> thus grow the community into this new field.
>> Many discussions recently about evolving the Flink runtime (both on the
>> mailing list and at conferences) show the interest in Flink users in the
>> space that Stateful Functions covers.
>> It seems natural that Stateful Functions should closely co-develop with
>> Apache Flink, ideally as part of the project.
>>
>> There are many details to be discusses, for example whether this should
>> be added to the Flink core repository, or whether we and to create a
>> separate repository
>> for this. But I think we should start discussing this after we have
>> consensus on whether the community wants this contribution.
>>
>> Really looking forward to hear what you think!
>>
>> Best Regards,
>> Stephan
>>
>>

Re: [PROPOSAL] Contribute Stateful Functions to Apache Flink

[Till Rohrmann <tr...@apache.org>]

Hi Stephan,

+1 for adding stateful functions to Flink. I believe the new set of
applications this feature will unlock will be super interesting for new and
existing Flink users alike.

One reason for not including it in the main repository would to not being
bound to Flink's release cadence. This would allow to release faster and
more often. However, I believe that having it eventually in Flink's main
repository would be beneficial in the long run.

Cheers,
Till

On Fri, Oct 11, 2019 at 12:56 PM Trevor Grant <tr...@gmail.com>
wrote:

> +1 non-binding on contribution.
>
> Separate repo, or feature branch to start maybe? I just feel like in the
> beginning this thing is going to have lots of breaking changes that maybe
> aren't going to fit well with tests / other "v1+" release code. Just my
> .02.
>
>
>
> On Fri, Oct 11, 2019 at 4:38 AM Stephan Ewen <se...@apache.org> wrote:
>
>> Dear Flink Community!
>>
>> Some of you probably heard it already: On Tuesday, at Flink Forward
>> Berlin, we announced **Stateful Functions**.
>>
>> Stateful Functions is a library on Flink to implement general purpose
>> applications. It is built around stateful functions (who would have thunk)
>> that can communicate arbitrarily through messages, have consistent state,
>> and a small resource footprint. They are a bit like keyed ProcessFunctions
>> that can send each other messages.
>> As simple as this sounds, this means you can now communicate in non-DAG
>> patterns, so it allows users to build programs they cannot build with Flink.
>> It also has other neat properties, like multiplexing of functions,
>> modular composition, tooling both container-based deployments and
>> as-a-Flink-job deployments.
>>
>> You can find out more about it here
>>   - Website: https://statefun.io/
>>   - Code: https://github.com/ververica/stateful-functions
>>   - Talk with motivation:
>> https://speakerdeck.com/stephanewen/stateful-functions-building-general-purpose-applications-and-services-on-apache-flink?slide=12
>>
>>
>> Now for the main issue: **We would like to contribute this project to
>> Apache Flink**
>>
>> I believe that this is a great fit for both sides.
>> For the Flink community, it would be a way to extend the capabilities and
>> use cases of Flink into a completely different type of applications and
>> thus grow the community into this new field.
>> Many discussions recently about evolving the Flink runtime (both on the
>> mailing list and at conferences) show the interest in Flink users in the
>> space that Stateful Functions covers.
>> It seems natural that Stateful Functions should closely co-develop with
>> Apache Flink, ideally as part of the project.
>>
>> There are many details to be discusses, for example whether this should
>> be added to the Flink core repository, or whether we and to create a
>> separate repository
>> for this. But I think we should start discussing this after we have
>> consensus on whether the community wants this contribution.
>>
>> Really looking forward to hear what you think!
>>
>> Best Regards,
>> Stephan
>>
>>

Re: [PROPOSAL] Contribute Stateful Functions to Apache Flink

[Trevor Grant <tr...@gmail.com>]

+1 non-binding on contribution.

Separate repo, or feature branch to start maybe? I just feel like in the
beginning this thing is going to have lots of breaking changes that maybe
aren't going to fit well with tests / other "v1+" release code. Just my
.02.



On Fri, Oct 11, 2019 at 4:38 AM Stephan Ewen <se...@apache.org> wrote:

> Dear Flink Community!
>
> Some of you probably heard it already: On Tuesday, at Flink Forward
> Berlin, we announced **Stateful Functions**.
>
> Stateful Functions is a library on Flink to implement general purpose
> applications. It is built around stateful functions (who would have thunk)
> that can communicate arbitrarily through messages, have consistent state,
> and a small resource footprint. They are a bit like keyed ProcessFunctions
> that can send each other messages.
> As simple as this sounds, this means you can now communicate in non-DAG
> patterns, so it allows users to build programs they cannot build with Flink.
> It also has other neat properties, like multiplexing of functions, modular
> composition, tooling both container-based deployments and as-a-Flink-job
> deployments.
>
> You can find out more about it here
>   - Website: https://statefun.io/
>   - Code: https://github.com/ververica/stateful-functions
>   - Talk with motivation:
> https://speakerdeck.com/stephanewen/stateful-functions-building-general-purpose-applications-and-services-on-apache-flink?slide=12
>
>
> Now for the main issue: **We would like to contribute this project to
> Apache Flink**
>
> I believe that this is a great fit for both sides.
> For the Flink community, it would be a way to extend the capabilities and
> use cases of Flink into a completely different type of applications and
> thus grow the community into this new field.
> Many discussions recently about evolving the Flink runtime (both on the
> mailing list and at conferences) show the interest in Flink users in the
> space that Stateful Functions covers.
> It seems natural that Stateful Functions should closely co-develop with
> Apache Flink, ideally as part of the project.
>
> There are many details to be discusses, for example whether this should be
> added to the Flink core repository, or whether we and to create a separate
> repository
> for this. But I think we should start discussing this after we have
> consensus on whether the community wants this contribution.
>
> Really looking forward to hear what you think!
>
> Best Regards,
> Stephan
>
>

Re: [PROPOSAL] Contribute Stateful Functions to Apache Flink

[Piotr Nowojski <pi...@ververica.com>]

Hi Stephan,

+1 for adding this to Apache Flink! 

Regarding the question if this should be committed to the main repository or as a separate one, I think it should be the main one. Previously we were discussing the idea of splitting Apache Flink into multiple repositories and I think the consensus was that it would cause more problems compared to the benefits.

Piotrek  

> On 11 Oct 2019, at 11:38, Stephan Ewen <se...@apache.org> wrote:
> 
> Dear Flink Community!
> 
> Some of you probably heard it already: On Tuesday, at Flink Forward Berlin, we announced **Stateful Functions**.
> 
> Stateful Functions is a library on Flink to implement general purpose applications. It is built around stateful functions (who would have thunk)
> that can communicate arbitrarily through messages, have consistent state, and a small resource footprint. They are a bit like keyed ProcessFunctions
> that can send each other messages.
> As simple as this sounds, this means you can now communicate in non-DAG patterns, so it allows users to build programs they cannot build with Flink.
> It also has other neat properties, like multiplexing of functions, modular composition, tooling both container-based deployments and as-a-Flink-job deployments.
> 
> You can find out more about it here
>   - Website: https://statefun.io/ <https://statefun.io/>
>   - Code: https://github.com/ververica/stateful-functions <https://github.com/ververica/stateful-functions>
>   - Talk with motivation: https://speakerdeck.com/stephanewen/stateful-functions-building-general-purpose-applications-and-services-on-apache-flink?slide=12 <https://speakerdeck.com/stephanewen/stateful-functions-building-general-purpose-applications-and-services-on-apache-flink?slide=12>
> 
> 
> Now for the main issue: **We would like to contribute this project to Apache Flink**
> 
> I believe that this is a great fit for both sides.
> For the Flink community, it would be a way to extend the capabilities and use cases of Flink into a completely different type of applications and thus grow the community into this new field.
> Many discussions recently about evolving the Flink runtime (both on the mailing list and at conferences) show the interest in Flink users in the space that Stateful Functions covers.
> It seems natural that Stateful Functions should closely co-develop with Apache Flink, ideally as part of the project.
> 
> There are many details to be discusses, for example whether this should be added to the Flink core repository, or whether we and to create a separate repository
> for this. But I think we should start discussing this after we have consensus on whether the community wants this contribution.
> 
> Really looking forward to hear what you think!
> 
> Best Regards,
> Stephan
> 

Re: [PROPOSAL] Contribute Stateful Functions to Apache Flink

[Trevor Grant <tr...@gmail.com>]

+1 non-binding on contribution.

Separate repo, or feature branch to start maybe? I just feel like in the
beginning this thing is going to have lots of breaking changes that maybe
aren't going to fit well with tests / other "v1+" release code. Just my
.02.



On Fri, Oct 11, 2019 at 4:38 AM Stephan Ewen <se...@apache.org> wrote:

> Dear Flink Community!
>
> Some of you probably heard it already: On Tuesday, at Flink Forward
> Berlin, we announced **Stateful Functions**.
>
> Stateful Functions is a library on Flink to implement general purpose
> applications. It is built around stateful functions (who would have thunk)
> that can communicate arbitrarily through messages, have consistent state,
> and a small resource footprint. They are a bit like keyed ProcessFunctions
> that can send each other messages.
> As simple as this sounds, this means you can now communicate in non-DAG
> patterns, so it allows users to build programs they cannot build with Flink.
> It also has other neat properties, like multiplexing of functions, modular
> composition, tooling both container-based deployments and as-a-Flink-job
> deployments.
>
> You can find out more about it here
>   - Website: https://statefun.io/
>   - Code: https://github.com/ververica/stateful-functions
>   - Talk with motivation:
> https://speakerdeck.com/stephanewen/stateful-functions-building-general-purpose-applications-and-services-on-apache-flink?slide=12
>
>
> Now for the main issue: **We would like to contribute this project to
> Apache Flink**
>
> I believe that this is a great fit for both sides.
> For the Flink community, it would be a way to extend the capabilities and
> use cases of Flink into a completely different type of applications and
> thus grow the community into this new field.
> Many discussions recently about evolving the Flink runtime (both on the
> mailing list and at conferences) show the interest in Flink users in the
> space that Stateful Functions covers.
> It seems natural that Stateful Functions should closely co-develop with
> Apache Flink, ideally as part of the project.
>
> There are many details to be discusses, for example whether this should be
> added to the Flink core repository, or whether we and to create a separate
> repository
> for this. But I think we should start discussing this after we have
> consensus on whether the community wants this contribution.
>
> Really looking forward to hear what you think!
>
> Best Regards,
> Stephan
>
>

Re: [PROPOSAL] Contribute Stateful Functions to Apache Flink

[Piotr Nowojski <pi...@ververica.com>]

Hi Stephan,

+1 for adding this to Apache Flink! 

Regarding the question if this should be committed to the main repository or as a separate one, I think it should be the main one. Previously we were discussing the idea of splitting Apache Flink into multiple repositories and I think the consensus was that it would cause more problems compared to the benefits.

Piotrek  

> On 11 Oct 2019, at 11:38, Stephan Ewen <se...@apache.org> wrote:
> 
> Dear Flink Community!
> 
> Some of you probably heard it already: On Tuesday, at Flink Forward Berlin, we announced **Stateful Functions**.
> 
> Stateful Functions is a library on Flink to implement general purpose applications. It is built around stateful functions (who would have thunk)
> that can communicate arbitrarily through messages, have consistent state, and a small resource footprint. They are a bit like keyed ProcessFunctions
> that can send each other messages.
> As simple as this sounds, this means you can now communicate in non-DAG patterns, so it allows users to build programs they cannot build with Flink.
> It also has other neat properties, like multiplexing of functions, modular composition, tooling both container-based deployments and as-a-Flink-job deployments.
> 
> You can find out more about it here
>   - Website: https://statefun.io/ <https://statefun.io/>
>   - Code: https://github.com/ververica/stateful-functions <https://github.com/ververica/stateful-functions>
>   - Talk with motivation: https://speakerdeck.com/stephanewen/stateful-functions-building-general-purpose-applications-and-services-on-apache-flink?slide=12 <https://speakerdeck.com/stephanewen/stateful-functions-building-general-purpose-applications-and-services-on-apache-flink?slide=12>
> 
> 
> Now for the main issue: **We would like to contribute this project to Apache Flink**
> 
> I believe that this is a great fit for both sides.
> For the Flink community, it would be a way to extend the capabilities and use cases of Flink into a completely different type of applications and thus grow the community into this new field.
> Many discussions recently about evolving the Flink runtime (both on the mailing list and at conferences) show the interest in Flink users in the space that Stateful Functions covers.
> It seems natural that Stateful Functions should closely co-develop with Apache Flink, ideally as part of the project.
> 
> There are many details to be discusses, for example whether this should be added to the Flink core repository, or whether we and to create a separate repository
> for this. But I think we should start discussing this after we have consensus on whether the community wants this contribution.
> 
> Really looking forward to hear what you think!
> 
> Best Regards,
> Stephan
>