You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flink.apache.org by "Tzu-Li (Gordon) Tai" <tz...@apache.org> on 2020/03/09 11:12:40 UTC
[DISCUSS] Update on Flink Stateful Functions & what are the next steps?
Hi devs,
I’d like to briefly provide an update / overview on what has been going on
in the Flink Stateful Functions project since it was moved to Apache Flink
[1].
One other main purpose of this is to figure out the next steps for the
project, such as release timing,
and some open questions surrounding updating the website, releasing
Stateful Functions Python artifacts and Docker images, etc.
*### Major new features*
Since the last Stateful Functions release (which back then was still
managed by Ververica, released roughly ~4 months ago), the community has
added a few noticeable new features:
Non-JVM language polyglot functions -
This was top priority on the roadmap, and was successfully added to
Stateful Functions over the past months.
Some major additions under this category include:
- Initial support for stateful polyglot functions, based on a HTTP-based
request-reply protocol.
- A convenience Python SDK that implements the request-reply protocol
(PR opened [2], still pending merge).
- Code-free way to bind functions / Kafka ingresses / Kafka egresses
using textual YAML-based modules.
Other noticeable additions -
- Synchronous test utility for users to unit test their stateful
functions
- Additional state primitives such as PersistedAppendingBuffer,
PersistedTable
- Stateful Function’s variant of the State Processor API, currently
supporting users to create Stateful Function application savepoints using
Java.
*### Release readiness*
There has also been some work in getting the project in shape and ready to
be released, both legal-wise and tooling-wise:
- Documents are migrated to be running on Apache infra [3] [4].
- LICENSES and NOTICES have been added for sources and artifacts with
bundled dependencies
- Release utility scripts to create release branches / create source
release / stage artifacts mostly adopted from the `apache/flink` repo.
- We’re still missing release process documentation specifically for
Stateful Functions in the community wiki, but it would make sense to add
this along the way during the first release.
*### Next steps?*
With the amount of changes and new features since the last release,
I think we’re at a good position to think about releasing the first version
managed by the Flink community soon.
What do you think?
There are also still some open questions that needs to be discussed, such
as:
- The previous website for the project before the project was move to
Apache Flink, https://statefun.io, still exists and points to the old
docs.
Should that be shutdown, or do we want to simply ask to have that
redirected to the new docs running on Apache infra?
- Do we want to update the Flink website’s main page navigation, so that
the Stateful Functions docs are navigable from the Flink website?
In general, how should the website look like if we add Stateful
Functions downloads / docs?
- Flink Stateful Functions will have Python packages that would be nice
to be installable using PyPI.
We can consider deploying that under the `pyflink` account [5], perhaps
under a new project, e.g. pypi.org/project/apache-flink-statefun?
- Flink Stateful Functions will also have a base Docker image that would
also be nice to be officially endorsed by the Flink community.
Would it make sense to incorporate this under the `apache/flink-docker`
repo [6]?
Please let us know any thoughts you may have!
Cheers,
Gordon
[1] https://github.com/apache/flink-statefun
[2] https://github.com/apache/flink-statefun/pull/52
[3] https://ci.apache.org/projects/flink/flink-statefun-docs-master/
[4]
https://ci.apache.org/projects/flink/flink-statefun-docs-master/api/java/
[5] https://pypi.org/user/pyflink/
[6] https://github.com/apache/flink-docker
Re: [DISCUSS] Update on Flink Stateful Functions & what are the next steps?
Posted by "Tzu-Li (Gordon) Tai" <tz...@apache.org>.
Hi,
Looks like that the general consent is to have a release now / very soon
for Flink Stateful Functions.
Since there are not objections so far, I'll proceed to kick off the release
(I'm volunteering to manage this release :)).
I'll cut off a feature branch, release-2.0, next Monday (Mar. 16th), and
aim to have RC1 out for flink-statefun 2.0.0 by end of next week (Mar.
20th).
As for how we want to incorporate Statefun Docker image releases in
apache/flink-docker, I'll start a separate discussion thread for that.
Cheers,
Gordon
On Tue, Mar 10, 2020 at 11:11 AM Tzu-Li (Gordon) Tai <tz...@apache.org>
wrote:
> Thanks for the feedback so far!
>
> @Stephan Ewen <se...@apache.org>
> I don't have an objection with 2.0 being the next version to release.
> My only concern is that at this point it's slightly hard to tell if the
> APIs are stable enough (especially the Python SDK),
> and it could be arguable that it is better to wait until we have more
> confidence before upticking the major version.
> But of course, that can also happen as yet another major version uptick
> (i.e. 3.0) if the API does end up changing a lot in the future.
> So, I don't have a strong opinion on this, in general I like the idea as
> it delivers a stronger message on the impact the new features has for
> applicable use cases.
>
> Thanks for starting a discussion thread on changes to the Flink website!
> Will be looking forward to that.
>
> On Mon, Mar 9, 2020 at 10:13 PM Seth Wiesman <sj...@gmail.com> wrote:
>
>> +1 to release.
>>
>> Stateful Functions has a strong set of core features, and a released
>> version will help drive adoption which will in turn help shape feature
>> development.
>>
>> Seth
>>
>> On Mon, Mar 9, 2020 at 6:59 AM Stephan Ewen <se...@apache.org> wrote:
>>
>> > Thanks, Gordon for this update.
>> >
>> > I think it would be great to do an Apache Release now.
>> > Given how much the "polyglot functions" (and the code-free
>> ingress/egress
>> > configuration) change the applicable use cases, I would actually call
>> this
>> > version 2.0. What do you think?
>> >
>> > About the https://statefun.io website: +1 to redirect this to the docs
>> > landing page.
>> >
>> > Also, +1 to link StateFun from the main Flink website. I'd be happy to
>> > start a discussion thread about this.
>> >
>> > Best,
>> > Stephan
>> >
>> >
>> >
>> >
>> > On Mon, Mar 9, 2020 at 12:34 PM Igal Shilman <ig...@ververica.com>
>> wrote:
>> >
>> > > Hi Gordon,
>> > > Thanks a lot for the summary and for kicking off the discussion!
>> > >
>> > > Regarding the next steps:
>> > >
>> > > * +1 to kick off the first release managed by the Flink community, as
>> the
>> > > amount of added features and fixies is substantial.
>> > > * +1 to have the docs centralized under the Apache infra and made
>> > > accessible via the main Flink docs.
>> > > * +1 to make the Python SDK installable via pip as a
>> > > "apache-flink-statefun" artifact.
>> > > * +1 to make the official Docker image accessible via Docker Hub by
>> > > incorporating them into apache/flink-docker repository.
>> > >
>> > > Cheers,
>> > > Igal.
>> > >
>> > > On Mon, Mar 9, 2020 at 12:12 PM Tzu-Li (Gordon) Tai <
>> tzulitai@apache.org
>> > >
>> > > wrote:
>> > >
>> > > > Hi devs,
>> > > >
>> > > > I’d like to briefly provide an update / overview on what has been
>> going
>> > > on
>> > > > in the Flink Stateful Functions project since it was moved to Apache
>> > > Flink
>> > > > [1].
>> > > >
>> > > > One other main purpose of this is to figure out the next steps for
>> the
>> > > > project, such as release timing,
>> > > > and some open questions surrounding updating the website, releasing
>> > > > Stateful Functions Python artifacts and Docker images, etc.
>> > > >
>> > > >
>> > > >
>> > > > *### Major new features*
>> > > > Since the last Stateful Functions release (which back then was still
>> > > > managed by Ververica, released roughly ~4 months ago), the community
>> > has
>> > > > added a few noticeable new features:
>> > > >
>> > > > Non-JVM language polyglot functions -
>> > > > This was top priority on the roadmap, and was successfully added to
>> > > > Stateful Functions over the past months.
>> > > > Some major additions under this category include:
>> > > >
>> > > > - Initial support for stateful polyglot functions, based on a
>> > > HTTP-based
>> > > > request-reply protocol.
>> > > > - A convenience Python SDK that implements the request-reply
>> > protocol
>> > > > (PR opened [2], still pending merge).
>> > > > - Code-free way to bind functions / Kafka ingresses / Kafka
>> egresses
>> > > > using textual YAML-based modules.
>> > > >
>> > > >
>> > > > Other noticeable additions -
>> > > >
>> > > > - Synchronous test utility for users to unit test their stateful
>> > > > functions
>> > > > - Additional state primitives such as PersistedAppendingBuffer,
>> > > > PersistedTable
>> > > > - Stateful Function’s variant of the State Processor API,
>> currently
>> > > > supporting users to create Stateful Function application
>> savepoints
>> > > > using
>> > > > Java.
>> > > >
>> > > >
>> > > >
>> > > > *### Release readiness*
>> > > > There has also been some work in getting the project in shape and
>> ready
>> > > to
>> > > > be released, both legal-wise and tooling-wise:
>> > > >
>> > > > - Documents are migrated to be running on Apache infra [3] [4].
>> > > > - LICENSES and NOTICES have been added for sources and artifacts
>> > with
>> > > > bundled dependencies
>> > > > - Release utility scripts to create release branches / create
>> source
>> > > > release / stage artifacts mostly adopted from the `apache/flink`
>> > repo.
>> > > > - We’re still missing release process documentation specifically
>> for
>> > > > Stateful Functions in the community wiki, but it would make
>> sense to
>> > > add
>> > > > this along the way during the first release.
>> > > >
>> > > >
>> > > > *### Next steps?*
>> > > >
>> > > > With the amount of changes and new features since the last release,
>> > > > I think we’re at a good position to think about releasing the first
>> > > version
>> > > > managed by the Flink community soon.
>> > > > What do you think?
>> > > >
>> > > > There are also still some open questions that needs to be discussed,
>> > such
>> > > > as:
>> > > >
>> > > > - The previous website for the project before the project was
>> move
>> > to
>> > > > Apache Flink, https://statefun.io, still exists and points to
>> the
>> > old
>> > > > docs.
>> > > > Should that be shutdown, or do we want to simply ask to have that
>> > > > redirected to the new docs running on Apache infra?
>> > > > - Do we want to update the Flink website’s main page navigation,
>> so
>> > > that
>> > > > the Stateful Functions docs are navigable from the Flink website?
>> > > > In general, how should the website look like if we add Stateful
>> > > > Functions downloads / docs?
>> > > > - Flink Stateful Functions will have Python packages that would
>> be
>> > > nice
>> > > > to be installable using PyPI.
>> > > > We can consider deploying that under the `pyflink` account [5],
>> > > perhaps
>> > > > under a new project, e.g. pypi.org/project/apache-flink-statefun
>> ?
>> > > > - Flink Stateful Functions will also have a base Docker image
>> that
>> > > would
>> > > > also be nice to be officially endorsed by the Flink community.
>> > > > Would it make sense to incorporate this under the
>> > > `apache/flink-docker`
>> > > > repo [6]?
>> > > >
>> > > >
>> > > > Please let us know any thoughts you may have!
>> > > >
>> > > > Cheers,
>> > > > Gordon
>> > > >
>> > > > [1] https://github.com/apache/flink-statefun
>> > > > [2] https://github.com/apache/flink-statefun/pull/52
>> > > > [3]
>> https://ci.apache.org/projects/flink/flink-statefun-docs-master/
>> > > > [4]
>> > > >
>> > >
>> >
>> https://ci.apache.org/projects/flink/flink-statefun-docs-master/api/java/
>> > > > [5] https://pypi.org/user/pyflink/
>> > > > [6] https://github.com/apache/flink-docker
>> > > >
>> > >
>> >
>>
>
Re: [DISCUSS] Update on Flink Stateful Functions & what are the next steps?
Posted by "Tzu-Li (Gordon) Tai" <tz...@apache.org>.
Thanks for the feedback so far!
@Stephan Ewen <se...@apache.org>
I don't have an objection with 2.0 being the next version to release.
My only concern is that at this point it's slightly hard to tell if the
APIs are stable enough (especially the Python SDK),
and it could be arguable that it is better to wait until we have more
confidence before upticking the major version.
But of course, that can also happen as yet another major version uptick
(i.e. 3.0) if the API does end up changing a lot in the future.
So, I don't have a strong opinion on this, in general I like the idea as it
delivers a stronger message on the impact the new features has for
applicable use cases.
Thanks for starting a discussion thread on changes to the Flink website!
Will be looking forward to that.
On Mon, Mar 9, 2020 at 10:13 PM Seth Wiesman <sj...@gmail.com> wrote:
> +1 to release.
>
> Stateful Functions has a strong set of core features, and a released
> version will help drive adoption which will in turn help shape feature
> development.
>
> Seth
>
> On Mon, Mar 9, 2020 at 6:59 AM Stephan Ewen <se...@apache.org> wrote:
>
> > Thanks, Gordon for this update.
> >
> > I think it would be great to do an Apache Release now.
> > Given how much the "polyglot functions" (and the code-free ingress/egress
> > configuration) change the applicable use cases, I would actually call
> this
> > version 2.0. What do you think?
> >
> > About the https://statefun.io website: +1 to redirect this to the docs
> > landing page.
> >
> > Also, +1 to link StateFun from the main Flink website. I'd be happy to
> > start a discussion thread about this.
> >
> > Best,
> > Stephan
> >
> >
> >
> >
> > On Mon, Mar 9, 2020 at 12:34 PM Igal Shilman <ig...@ververica.com> wrote:
> >
> > > Hi Gordon,
> > > Thanks a lot for the summary and for kicking off the discussion!
> > >
> > > Regarding the next steps:
> > >
> > > * +1 to kick off the first release managed by the Flink community, as
> the
> > > amount of added features and fixies is substantial.
> > > * +1 to have the docs centralized under the Apache infra and made
> > > accessible via the main Flink docs.
> > > * +1 to make the Python SDK installable via pip as a
> > > "apache-flink-statefun" artifact.
> > > * +1 to make the official Docker image accessible via Docker Hub by
> > > incorporating them into apache/flink-docker repository.
> > >
> > > Cheers,
> > > Igal.
> > >
> > > On Mon, Mar 9, 2020 at 12:12 PM Tzu-Li (Gordon) Tai <
> tzulitai@apache.org
> > >
> > > wrote:
> > >
> > > > Hi devs,
> > > >
> > > > I’d like to briefly provide an update / overview on what has been
> going
> > > on
> > > > in the Flink Stateful Functions project since it was moved to Apache
> > > Flink
> > > > [1].
> > > >
> > > > One other main purpose of this is to figure out the next steps for
> the
> > > > project, such as release timing,
> > > > and some open questions surrounding updating the website, releasing
> > > > Stateful Functions Python artifacts and Docker images, etc.
> > > >
> > > >
> > > >
> > > > *### Major new features*
> > > > Since the last Stateful Functions release (which back then was still
> > > > managed by Ververica, released roughly ~4 months ago), the community
> > has
> > > > added a few noticeable new features:
> > > >
> > > > Non-JVM language polyglot functions -
> > > > This was top priority on the roadmap, and was successfully added to
> > > > Stateful Functions over the past months.
> > > > Some major additions under this category include:
> > > >
> > > > - Initial support for stateful polyglot functions, based on a
> > > HTTP-based
> > > > request-reply protocol.
> > > > - A convenience Python SDK that implements the request-reply
> > protocol
> > > > (PR opened [2], still pending merge).
> > > > - Code-free way to bind functions / Kafka ingresses / Kafka
> egresses
> > > > using textual YAML-based modules.
> > > >
> > > >
> > > > Other noticeable additions -
> > > >
> > > > - Synchronous test utility for users to unit test their stateful
> > > > functions
> > > > - Additional state primitives such as PersistedAppendingBuffer,
> > > > PersistedTable
> > > > - Stateful Function’s variant of the State Processor API,
> currently
> > > > supporting users to create Stateful Function application
> savepoints
> > > > using
> > > > Java.
> > > >
> > > >
> > > >
> > > > *### Release readiness*
> > > > There has also been some work in getting the project in shape and
> ready
> > > to
> > > > be released, both legal-wise and tooling-wise:
> > > >
> > > > - Documents are migrated to be running on Apache infra [3] [4].
> > > > - LICENSES and NOTICES have been added for sources and artifacts
> > with
> > > > bundled dependencies
> > > > - Release utility scripts to create release branches / create
> source
> > > > release / stage artifacts mostly adopted from the `apache/flink`
> > repo.
> > > > - We’re still missing release process documentation specifically
> for
> > > > Stateful Functions in the community wiki, but it would make sense
> to
> > > add
> > > > this along the way during the first release.
> > > >
> > > >
> > > > *### Next steps?*
> > > >
> > > > With the amount of changes and new features since the last release,
> > > > I think we’re at a good position to think about releasing the first
> > > version
> > > > managed by the Flink community soon.
> > > > What do you think?
> > > >
> > > > There are also still some open questions that needs to be discussed,
> > such
> > > > as:
> > > >
> > > > - The previous website for the project before the project was move
> > to
> > > > Apache Flink, https://statefun.io, still exists and points to the
> > old
> > > > docs.
> > > > Should that be shutdown, or do we want to simply ask to have that
> > > > redirected to the new docs running on Apache infra?
> > > > - Do we want to update the Flink website’s main page navigation,
> so
> > > that
> > > > the Stateful Functions docs are navigable from the Flink website?
> > > > In general, how should the website look like if we add Stateful
> > > > Functions downloads / docs?
> > > > - Flink Stateful Functions will have Python packages that would be
> > > nice
> > > > to be installable using PyPI.
> > > > We can consider deploying that under the `pyflink` account [5],
> > > perhaps
> > > > under a new project, e.g. pypi.org/project/apache-flink-statefun?
> > > > - Flink Stateful Functions will also have a base Docker image that
> > > would
> > > > also be nice to be officially endorsed by the Flink community.
> > > > Would it make sense to incorporate this under the
> > > `apache/flink-docker`
> > > > repo [6]?
> > > >
> > > >
> > > > Please let us know any thoughts you may have!
> > > >
> > > > Cheers,
> > > > Gordon
> > > >
> > > > [1] https://github.com/apache/flink-statefun
> > > > [2] https://github.com/apache/flink-statefun/pull/52
> > > > [3] https://ci.apache.org/projects/flink/flink-statefun-docs-master/
> > > > [4]
> > > >
> > >
> >
> https://ci.apache.org/projects/flink/flink-statefun-docs-master/api/java/
> > > > [5] https://pypi.org/user/pyflink/
> > > > [6] https://github.com/apache/flink-docker
> > > >
> > >
> >
>
Re: [DISCUSS] Update on Flink Stateful Functions & what are the next steps?
Posted by Seth Wiesman <sj...@gmail.com>.
+1 to release.
Stateful Functions has a strong set of core features, and a released
version will help drive adoption which will in turn help shape feature
development.
Seth
On Mon, Mar 9, 2020 at 6:59 AM Stephan Ewen <se...@apache.org> wrote:
> Thanks, Gordon for this update.
>
> I think it would be great to do an Apache Release now.
> Given how much the "polyglot functions" (and the code-free ingress/egress
> configuration) change the applicable use cases, I would actually call this
> version 2.0. What do you think?
>
> About the https://statefun.io website: +1 to redirect this to the docs
> landing page.
>
> Also, +1 to link StateFun from the main Flink website. I'd be happy to
> start a discussion thread about this.
>
> Best,
> Stephan
>
>
>
>
> On Mon, Mar 9, 2020 at 12:34 PM Igal Shilman <ig...@ververica.com> wrote:
>
> > Hi Gordon,
> > Thanks a lot for the summary and for kicking off the discussion!
> >
> > Regarding the next steps:
> >
> > * +1 to kick off the first release managed by the Flink community, as the
> > amount of added features and fixies is substantial.
> > * +1 to have the docs centralized under the Apache infra and made
> > accessible via the main Flink docs.
> > * +1 to make the Python SDK installable via pip as a
> > "apache-flink-statefun" artifact.
> > * +1 to make the official Docker image accessible via Docker Hub by
> > incorporating them into apache/flink-docker repository.
> >
> > Cheers,
> > Igal.
> >
> > On Mon, Mar 9, 2020 at 12:12 PM Tzu-Li (Gordon) Tai <tzulitai@apache.org
> >
> > wrote:
> >
> > > Hi devs,
> > >
> > > I’d like to briefly provide an update / overview on what has been going
> > on
> > > in the Flink Stateful Functions project since it was moved to Apache
> > Flink
> > > [1].
> > >
> > > One other main purpose of this is to figure out the next steps for the
> > > project, such as release timing,
> > > and some open questions surrounding updating the website, releasing
> > > Stateful Functions Python artifacts and Docker images, etc.
> > >
> > >
> > >
> > > *### Major new features*
> > > Since the last Stateful Functions release (which back then was still
> > > managed by Ververica, released roughly ~4 months ago), the community
> has
> > > added a few noticeable new features:
> > >
> > > Non-JVM language polyglot functions -
> > > This was top priority on the roadmap, and was successfully added to
> > > Stateful Functions over the past months.
> > > Some major additions under this category include:
> > >
> > > - Initial support for stateful polyglot functions, based on a
> > HTTP-based
> > > request-reply protocol.
> > > - A convenience Python SDK that implements the request-reply
> protocol
> > > (PR opened [2], still pending merge).
> > > - Code-free way to bind functions / Kafka ingresses / Kafka egresses
> > > using textual YAML-based modules.
> > >
> > >
> > > Other noticeable additions -
> > >
> > > - Synchronous test utility for users to unit test their stateful
> > > functions
> > > - Additional state primitives such as PersistedAppendingBuffer,
> > > PersistedTable
> > > - Stateful Function’s variant of the State Processor API, currently
> > > supporting users to create Stateful Function application savepoints
> > > using
> > > Java.
> > >
> > >
> > >
> > > *### Release readiness*
> > > There has also been some work in getting the project in shape and ready
> > to
> > > be released, both legal-wise and tooling-wise:
> > >
> > > - Documents are migrated to be running on Apache infra [3] [4].
> > > - LICENSES and NOTICES have been added for sources and artifacts
> with
> > > bundled dependencies
> > > - Release utility scripts to create release branches / create source
> > > release / stage artifacts mostly adopted from the `apache/flink`
> repo.
> > > - We’re still missing release process documentation specifically for
> > > Stateful Functions in the community wiki, but it would make sense to
> > add
> > > this along the way during the first release.
> > >
> > >
> > > *### Next steps?*
> > >
> > > With the amount of changes and new features since the last release,
> > > I think we’re at a good position to think about releasing the first
> > version
> > > managed by the Flink community soon.
> > > What do you think?
> > >
> > > There are also still some open questions that needs to be discussed,
> such
> > > as:
> > >
> > > - The previous website for the project before the project was move
> to
> > > Apache Flink, https://statefun.io, still exists and points to the
> old
> > > docs.
> > > Should that be shutdown, or do we want to simply ask to have that
> > > redirected to the new docs running on Apache infra?
> > > - Do we want to update the Flink website’s main page navigation, so
> > that
> > > the Stateful Functions docs are navigable from the Flink website?
> > > In general, how should the website look like if we add Stateful
> > > Functions downloads / docs?
> > > - Flink Stateful Functions will have Python packages that would be
> > nice
> > > to be installable using PyPI.
> > > We can consider deploying that under the `pyflink` account [5],
> > perhaps
> > > under a new project, e.g. pypi.org/project/apache-flink-statefun?
> > > - Flink Stateful Functions will also have a base Docker image that
> > would
> > > also be nice to be officially endorsed by the Flink community.
> > > Would it make sense to incorporate this under the
> > `apache/flink-docker`
> > > repo [6]?
> > >
> > >
> > > Please let us know any thoughts you may have!
> > >
> > > Cheers,
> > > Gordon
> > >
> > > [1] https://github.com/apache/flink-statefun
> > > [2] https://github.com/apache/flink-statefun/pull/52
> > > [3] https://ci.apache.org/projects/flink/flink-statefun-docs-master/
> > > [4]
> > >
> >
> https://ci.apache.org/projects/flink/flink-statefun-docs-master/api/java/
> > > [5] https://pypi.org/user/pyflink/
> > > [6] https://github.com/apache/flink-docker
> > >
> >
>
Re: [DISCUSS] Update on Flink Stateful Functions & what are the next steps?
Posted by Stephan Ewen <se...@apache.org>.
Thanks, Gordon for this update.
I think it would be great to do an Apache Release now.
Given how much the "polyglot functions" (and the code-free ingress/egress
configuration) change the applicable use cases, I would actually call this
version 2.0. What do you think?
About the https://statefun.io website: +1 to redirect this to the docs
landing page.
Also, +1 to link StateFun from the main Flink website. I'd be happy to
start a discussion thread about this.
Best,
Stephan
On Mon, Mar 9, 2020 at 12:34 PM Igal Shilman <ig...@ververica.com> wrote:
> Hi Gordon,
> Thanks a lot for the summary and for kicking off the discussion!
>
> Regarding the next steps:
>
> * +1 to kick off the first release managed by the Flink community, as the
> amount of added features and fixies is substantial.
> * +1 to have the docs centralized under the Apache infra and made
> accessible via the main Flink docs.
> * +1 to make the Python SDK installable via pip as a
> "apache-flink-statefun" artifact.
> * +1 to make the official Docker image accessible via Docker Hub by
> incorporating them into apache/flink-docker repository.
>
> Cheers,
> Igal.
>
> On Mon, Mar 9, 2020 at 12:12 PM Tzu-Li (Gordon) Tai <tz...@apache.org>
> wrote:
>
> > Hi devs,
> >
> > I’d like to briefly provide an update / overview on what has been going
> on
> > in the Flink Stateful Functions project since it was moved to Apache
> Flink
> > [1].
> >
> > One other main purpose of this is to figure out the next steps for the
> > project, such as release timing,
> > and some open questions surrounding updating the website, releasing
> > Stateful Functions Python artifacts and Docker images, etc.
> >
> >
> >
> > *### Major new features*
> > Since the last Stateful Functions release (which back then was still
> > managed by Ververica, released roughly ~4 months ago), the community has
> > added a few noticeable new features:
> >
> > Non-JVM language polyglot functions -
> > This was top priority on the roadmap, and was successfully added to
> > Stateful Functions over the past months.
> > Some major additions under this category include:
> >
> > - Initial support for stateful polyglot functions, based on a
> HTTP-based
> > request-reply protocol.
> > - A convenience Python SDK that implements the request-reply protocol
> > (PR opened [2], still pending merge).
> > - Code-free way to bind functions / Kafka ingresses / Kafka egresses
> > using textual YAML-based modules.
> >
> >
> > Other noticeable additions -
> >
> > - Synchronous test utility for users to unit test their stateful
> > functions
> > - Additional state primitives such as PersistedAppendingBuffer,
> > PersistedTable
> > - Stateful Function’s variant of the State Processor API, currently
> > supporting users to create Stateful Function application savepoints
> > using
> > Java.
> >
> >
> >
> > *### Release readiness*
> > There has also been some work in getting the project in shape and ready
> to
> > be released, both legal-wise and tooling-wise:
> >
> > - Documents are migrated to be running on Apache infra [3] [4].
> > - LICENSES and NOTICES have been added for sources and artifacts with
> > bundled dependencies
> > - Release utility scripts to create release branches / create source
> > release / stage artifacts mostly adopted from the `apache/flink` repo.
> > - We’re still missing release process documentation specifically for
> > Stateful Functions in the community wiki, but it would make sense to
> add
> > this along the way during the first release.
> >
> >
> > *### Next steps?*
> >
> > With the amount of changes and new features since the last release,
> > I think we’re at a good position to think about releasing the first
> version
> > managed by the Flink community soon.
> > What do you think?
> >
> > There are also still some open questions that needs to be discussed, such
> > as:
> >
> > - The previous website for the project before the project was move to
> > Apache Flink, https://statefun.io, still exists and points to the old
> > docs.
> > Should that be shutdown, or do we want to simply ask to have that
> > redirected to the new docs running on Apache infra?
> > - Do we want to update the Flink website’s main page navigation, so
> that
> > the Stateful Functions docs are navigable from the Flink website?
> > In general, how should the website look like if we add Stateful
> > Functions downloads / docs?
> > - Flink Stateful Functions will have Python packages that would be
> nice
> > to be installable using PyPI.
> > We can consider deploying that under the `pyflink` account [5],
> perhaps
> > under a new project, e.g. pypi.org/project/apache-flink-statefun?
> > - Flink Stateful Functions will also have a base Docker image that
> would
> > also be nice to be officially endorsed by the Flink community.
> > Would it make sense to incorporate this under the
> `apache/flink-docker`
> > repo [6]?
> >
> >
> > Please let us know any thoughts you may have!
> >
> > Cheers,
> > Gordon
> >
> > [1] https://github.com/apache/flink-statefun
> > [2] https://github.com/apache/flink-statefun/pull/52
> > [3] https://ci.apache.org/projects/flink/flink-statefun-docs-master/
> > [4]
> >
> https://ci.apache.org/projects/flink/flink-statefun-docs-master/api/java/
> > [5] https://pypi.org/user/pyflink/
> > [6] https://github.com/apache/flink-docker
> >
>
Re: [DISCUSS] Update on Flink Stateful Functions & what are the next steps?
Posted by Igal Shilman <ig...@ververica.com>.
Hi Gordon,
Thanks a lot for the summary and for kicking off the discussion!
Regarding the next steps:
* +1 to kick off the first release managed by the Flink community, as the
amount of added features and fixies is substantial.
* +1 to have the docs centralized under the Apache infra and made
accessible via the main Flink docs.
* +1 to make the Python SDK installable via pip as a
"apache-flink-statefun" artifact.
* +1 to make the official Docker image accessible via Docker Hub by
incorporating them into apache/flink-docker repository.
Cheers,
Igal.
On Mon, Mar 9, 2020 at 12:12 PM Tzu-Li (Gordon) Tai <tz...@apache.org>
wrote:
> Hi devs,
>
> I’d like to briefly provide an update / overview on what has been going on
> in the Flink Stateful Functions project since it was moved to Apache Flink
> [1].
>
> One other main purpose of this is to figure out the next steps for the
> project, such as release timing,
> and some open questions surrounding updating the website, releasing
> Stateful Functions Python artifacts and Docker images, etc.
>
>
>
> *### Major new features*
> Since the last Stateful Functions release (which back then was still
> managed by Ververica, released roughly ~4 months ago), the community has
> added a few noticeable new features:
>
> Non-JVM language polyglot functions -
> This was top priority on the roadmap, and was successfully added to
> Stateful Functions over the past months.
> Some major additions under this category include:
>
> - Initial support for stateful polyglot functions, based on a HTTP-based
> request-reply protocol.
> - A convenience Python SDK that implements the request-reply protocol
> (PR opened [2], still pending merge).
> - Code-free way to bind functions / Kafka ingresses / Kafka egresses
> using textual YAML-based modules.
>
>
> Other noticeable additions -
>
> - Synchronous test utility for users to unit test their stateful
> functions
> - Additional state primitives such as PersistedAppendingBuffer,
> PersistedTable
> - Stateful Function’s variant of the State Processor API, currently
> supporting users to create Stateful Function application savepoints
> using
> Java.
>
>
>
> *### Release readiness*
> There has also been some work in getting the project in shape and ready to
> be released, both legal-wise and tooling-wise:
>
> - Documents are migrated to be running on Apache infra [3] [4].
> - LICENSES and NOTICES have been added for sources and artifacts with
> bundled dependencies
> - Release utility scripts to create release branches / create source
> release / stage artifacts mostly adopted from the `apache/flink` repo.
> - We’re still missing release process documentation specifically for
> Stateful Functions in the community wiki, but it would make sense to add
> this along the way during the first release.
>
>
> *### Next steps?*
>
> With the amount of changes and new features since the last release,
> I think we’re at a good position to think about releasing the first version
> managed by the Flink community soon.
> What do you think?
>
> There are also still some open questions that needs to be discussed, such
> as:
>
> - The previous website for the project before the project was move to
> Apache Flink, https://statefun.io, still exists and points to the old
> docs.
> Should that be shutdown, or do we want to simply ask to have that
> redirected to the new docs running on Apache infra?
> - Do we want to update the Flink website’s main page navigation, so that
> the Stateful Functions docs are navigable from the Flink website?
> In general, how should the website look like if we add Stateful
> Functions downloads / docs?
> - Flink Stateful Functions will have Python packages that would be nice
> to be installable using PyPI.
> We can consider deploying that under the `pyflink` account [5], perhaps
> under a new project, e.g. pypi.org/project/apache-flink-statefun?
> - Flink Stateful Functions will also have a base Docker image that would
> also be nice to be officially endorsed by the Flink community.
> Would it make sense to incorporate this under the `apache/flink-docker`
> repo [6]?
>
>
> Please let us know any thoughts you may have!
>
> Cheers,
> Gordon
>
> [1] https://github.com/apache/flink-statefun
> [2] https://github.com/apache/flink-statefun/pull/52
> [3] https://ci.apache.org/projects/flink/flink-statefun-docs-master/
> [4]
> https://ci.apache.org/projects/flink/flink-statefun-docs-master/api/java/
> [5] https://pypi.org/user/pyflink/
> [6] https://github.com/apache/flink-docker
>