You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flink.apache.org by Filip Karnicki <fi...@gmail.com> on 2022/10/26 10:29:36 UTC
Stateful Functions with Flink 1.15 and onwards
Hi, I noticed that the development on stateful functions has come to a bit
of a halt, with a pull request to update statefun to use Flink 1.15 being
in the `open` state since May 2022.
What do we think is the future of this sub-project?
The background to this question is that my team is on a shared Flink
cluster which will soon be upgrading to Flink 1.15. If I need to re-write
all our code as a native Flink job (rather than a remote stateful function)
then I need to get started right away.
Many thanks,
Fil
Re: Stateful Functions with Flink 1.15 and onwards
Posted by Maximilian Michels <mx...@apache.org>.
As far as I know the original authors are not working on this
implementation anymore. Other folks might be able to provide more context.
Instead of rewriting your code to be a native Flink job, would it be an
option for your team to pick up the pull request to port statefun to Flink
1.15?
-Max
On Wed, Oct 26, 2022 at 3:31 AM Filip Karnicki <fi...@gmail.com>
wrote:
> Hi, I noticed that the development on stateful functions has come to a bit
> of a halt, with a pull request to update statefun to use Flink 1.15 being
> in the `open` state since May 2022.
>
> What do we think is the future of this sub-project?
>
> The background to this question is that my team is on a shared Flink
> cluster which will soon be upgrading to Flink 1.15. If I need to re-write
> all our code as a native Flink job (rather than a remote stateful function)
> then I need to get started right away.
>
> Many thanks,
> Fil
>
Re: Stateful Functions with Flink 1.15 and onwards
Posted by Galen Warren <ga...@cvillewarrens.com>.
I thought I'd check in again here, is there any update? If there's anything
I can do to help with releasing statefun 3.3, just let me know. Thanks.
On Mon, Nov 28, 2022 at 8:12 AM Galen Warren <ga...@cvillewarrens.com>
wrote:
> Hi Gordon, just checking in. Is there anything I can do to help here?
>
> On Fri, Nov 4, 2022 at 12:59 PM Tzu-Li (Gordon) Tai <tz...@apache.org>
> wrote:
>
>> @Galen The next step is essentially for someone to be the release manager
>> and drive the release process for StateFun 3.3.0 [1]. The document for the
>> release process might be slightly outdated in some places, but overall
>> outlines the process pretty clearly.
>>
>> As I mentioned earlier, there's quite a few steps in the process that
>> requires committer write access, so it would be easier if a committer can
>> pick this up as the release manager.
>> I'll be happy to take this on, but I'll only have availability after next
>> week. If someone else is willing to take this on earlier it would
>> certainly
>> be better to get the release out ASAP.
>>
>> [1]
>>
>> https://cwiki.apache.org/confluence/display/FLINK/Creating+a+Flink+Stateful+Functions+Release
>>
>> On Fri, Nov 4, 2022 at 5:54 AM Galen Warren <ga...@cvillewarrens.com>
>> wrote:
>>
>> > Thanks Gordon.
>> >
>> > What is the next step here?
>> >
>> > On Thu, Nov 3, 2022 at 1:45 PM Tzu-Li (Gordon) Tai <tzulitai@apache.org
>> >
>> > wrote:
>> >
>> > > FYI, release-3.3 branch has been cut and is ready for the release
>> process
>> > > for StateFun 3.3.0:
>> > > https://github.com/apache/flink-statefun/tree/release-3.3
>> > >
>> > > On Tue, Nov 1, 2022 at 10:21 AM Tzu-Li (Gordon) Tai <
>> tzulitai@apache.org
>> > >
>> > > wrote:
>> > >
>> > >> Btw, I'll assume that we're using this thread to gather consensus for
>> > >> code-freezing for 3.3.x series of StateFun. I know there hasn't been
>> > much
>> > >> activity on the repo, so this is just a formality really :)
>> > >>
>> > >> From the commit history, it looks like we're mainly including the
>> below
>> > >> major changes and bug fixes for 3.3.x:
>> > >> - Flink upgrade to 1.15.2
>> > >> - https://issues.apache.org/jira/browse/FLINK-26340
>> > >> - https://issues.apache.org/jira/browse/FLINK-25866
>> > >> - https://issues.apache.org/jira/browse/FLINK-25936
>> > >> - https://issues.apache.org/jira/browse/FLINK-25933
>> > >>
>> > >> I'll wait for 24 hours before cutting the release branch for 3.3.x,
>> > >> unless anyone raises any objections before that.
>> > >>
>> > >> Thanks,
>> > >> Gordon
>> > >>
>> > >> On Tue, Nov 1, 2022 at 10:09 AM Galen Warren <
>> galen@cvillewarrens.com>
>> > >> wrote:
>> > >>
>> > >>> Thanks Gordon and Filip, I appreciate your help on this one.
>> > >>>
>> > >>> On Tue, Nov 1, 2022 at 1:07 PM Tzu-Li (Gordon) Tai <
>> > tzulitai@apache.org>
>> > >>> wrote:
>> > >>>
>> > >>>> PR for upgrading to Flink 1.15.2 has been merged. Thanks for the
>> > >>>> efforts,
>> > >>>> Galen and Filip!
>> > >>>>
>> > >>>> We should be ready to kick off a new release for StateFun with the
>> > Flink
>> > >>>> version upgrade.
>> > >>>> I'll cut off a release branch now on apache/flink-statefun for
>> > >>>> release-3.3.x to move things forward.
>> > >>>> @Galen, @Filip if you want to, after the release branch is cut, you
>> > >>>> could
>> > >>>> probably upgrade the master branch to Flink 1.16.x as well.
>> > >>>>
>> > >>>> Afterwards we should decide who is available to drive the actual
>> > release
>> > >>>> process for 3.3.0.
>> > >>>> There's quite a few steps that would require committer write
>> access.
>> > >>>> Unless someone else is up for this earlier, I'll have some
>> > availability
>> > >>>> towards the end of next week to help drive this.
>> > >>>>
>> > >>>> Thanks,
>> > >>>> Gordon
>> > >>>>
>> > >>>> On Mon, Oct 31, 2022 at 12:17 PM Galen Warren <
>> > galen@cvillewarrens.com>
>> > >>>> wrote:
>> > >>>>
>> > >>>> > Yes, that makes sense.
>> > >>>> >
>> > >>>> > PR is here: [FLINK-29814][statefun] Change supported Flink
>> version
>> > to
>> > >>>> > 1.15.2 by galenwarren · Pull Request #319 · apache/flink-statefun
>> > >>>> > (github.com) <https://github.com/apache/flink-statefun/pull/319
>> >.
>> > >>>> >
>> > >>>> > On Mon, Oct 31, 2022 at 11:35 AM Till Rohrmann <
>> > trohrmann@apache.org>
>> > >>>> > wrote:
>> > >>>> >
>> > >>>> > > I think there might still be value in supporting 1.15 since not
>> > >>>> everyone
>> > >>>> > > upgrades Flink very fast. Hopefully, for Statefun the diff
>> between
>> > >>>> Flink
>> > >>>> > > 1.15 and 1.16 boils down to changing the Flink dependencies.
>> > >>>> > >
>> > >>>> > > Cheers,
>> > >>>> > > Till
>> > >>>> > >
>> > >>>> > > On Mon, Oct 31, 2022 at 2:06 PM Galen Warren <
>> > >>>> galen@cvillewarrens.com>
>> > >>>> > > wrote:
>> > >>>> > >
>> > >>>> > >> Sure thing. One question -- Flink 1.16 was just released a few
>> > >>>> days ago.
>> > >>>> > >> Should I support 1.15, or just go straight to 1.16?
>> > >>>> > >>
>> > >>>> > >> On Mon, Oct 31, 2022 at 8:49 AM Till Rohrmann <
>> > >>>> trohrmann@apache.org>
>> > >>>> > >> wrote:
>> > >>>> > >>
>> > >>>> > >>> Hi folks,
>> > >>>> > >>>
>> > >>>> > >>> if you can open a PR for supporting Flink 1.15 Galen, then
>> this
>> > >>>> would
>> > >>>> > be
>> > >>>> > >>> awesome. I've assigned you to this ticket. The next thing
>> after
>> > >>>> merging
>> > >>>> > >>> this PR would be creating a new StateFun release. Once we
>> have
>> > >>>> merged
>> > >>>> > the
>> > >>>> > >>> PR, let's check who can help with it the fastest.
>> > >>>> > >>>
>> > >>>> > >>> Cheers,
>> > >>>> > >>> Till
>> > >>>> > >>>
>> > >>>> > >>> On Mon, Oct 31, 2022 at 1:10 PM Galen Warren <
>> > >>>> galen@cvillewarrens.com>
>> > >>>> > >>> wrote:
>> > >>>> > >>>
>> > >>>> > >>>> Yes, I could do that.
>> > >>>> > >>>>
>> > >>>> > >>>> On Mon, Oct 31, 2022 at 7:48 AM Filip Karnicki <
>> > >>>> > >>>> filip.karnicki@gmail.com> wrote:
>> > >>>> > >>>>
>> > >>>> > >>>>> Hi All
>> > >>>> > >>>>>
>> > >>>> > >>>>> So what's the play here?
>> > >>>> > >>>>>
>> > >>>> > >>>>> Galen, what do you think about taking this on? Perhaps
>> ++Till
>> > >>>> would
>> > >>>> > >>>>> assign this jira to you (with your permission) given he's
>> > >>>> helped me
>> > >>>> > out
>> > >>>> > >>>>> with statefun work before
>> > >>>> > >>>>> https://issues.apache.org/jira/browse/FLINK-29814
>> > >>>> > >>>>>
>> > >>>> > >>>>> I can try to move to move statefun to flink 1.16 when it's
>> out
>> > >>>> > >>>>>
>> > >>>> > >>>>>
>> > >>>> > >>>>> Kind regards
>> > >>>> > >>>>> Fil
>> > >>>> > >>>>>
>> > >>>> > >>>>> On Thu, 27 Oct 2022 at 10:02, Filip Karnicki <
>> > >>>> > filip.karnicki@gmail.com>
>> > >>>> > >>>>> wrote:
>> > >>>> > >>>>>
>> > >>>> > >>>>>> Hi All
>> > >>>> > >>>>>>
>> > >>>> > >>>>>> Our use case is that we need to process elements for the
>> same
>> > >>>> key
>> > >>>> > >>>>>> sequentially, and this processing involves async
>> operations.
>> > >>>> > >>>>>>
>> > >>>> > >>>>>> If any part of the processing fails, we store the
>> offending
>> > >>>> and all
>> > >>>> > >>>>>> subsequent incoming messages for that key in the state and
>> > not
>> > >>>> > process any
>> > >>>> > >>>>>> further messages for that key, until a retry succeeds or a
>> > >>>> human
>> > >>>> > sends a
>> > >>>> > >>>>>> 'skip' command message.
>> > >>>> > >>>>>>
>> > >>>> > >>>>>> diagram:
>> > >>>> > >>>>>>
>> > >>>> >
>> > >>>>
>> >
>> https://mermaid.live/edit#pako:eNplkL1uwzAMhF-F0JQADrp76FR06tSOcQfWom3V-nFFqoUR591L20mXaqAOxHd3AC-mTZZMbfqM0wAvr00EfS62KbjYn-8C6Jui8DucTo_wmT4Oz97FEVhQqCtxXR13q_IBo-XzXWyehUc3LSu2Uyq2qIXpq2iyQ-9nmCjDSPMCmUISOuwfaEErLsVbw2272VOEDp0vmSqw5HEmC4GYsSeQpKjkv7j_buQ5tjAV4YeehOHHyQDsLAF1HbXCCyQZKB-2CTyzUOCjqUygHNBZPdxljW2MAoEaU6u0mMfGNPGqHBZJb1piasmFKlMmqxd7cqj3Dqbu0DNdfwHTGoek
>> > >>>> > >>>>>> mermaid (in case mermaid.live goes down in the future):
>> > >>>> > >>>>>> graph LR
>> > >>>> > >>>>>> incoming[incoming events] --> job(Flink statefun job)
>> > >>>> > >>>>>> commands[commands] -->|skip| job
>> > >>>> > >>>>>> job --> |sequentially per key| remote(remote function)
>> > >>>> > >>>>>> remote --> |on failure, delayed message to retry|
>> remote
>> > >>>> > >>>>>> remote --> |async puts/gets with side effects|
>> > other(other
>> > >>>> > >>>>>> systems)
>> > >>>> > >>>>>>
>> > >>>> > >>>>>> Having the processing happen outside of Flink is
>> nice-to-have
>> > >>>> from
>> > >>>> > an
>> > >>>> > >>>>>> independent scalability point of view, but is not strictly
>> > >>>> required.
>> > >>>> > >>>>>>
>> > >>>> > >>>>>> So long story short - no cyclic messaging, but also no
>> way I
>> > >>>> can
>> > >>>> > >>>>>> think of to use existing native Flink operators like async
>> > i/o
>> > >>>> > (which when
>> > >>>> > >>>>>> I last checked a few years back didn't have access to
>> keyed
>> > >>>> state)
>> > >>>> > >>>>>>
>> > >>>> > >>>>>>
>> > >>>> > >>>>>> P.S. Please note that there is already a pull request that
>> > has
>> > >>>> > >>>>>> something to do wtih Flink 1.15, albeit without a
>> description
>> > >>>> or a
>> > >>>> > jira:
>> > >>>> > >>>>>> https://github.com/apache/flink-statefun/pull/314
>> > >>>> > >>>>>>
>> > >>>> > >>>>>>
>> > >>>> > >>>>>> On Wed, 26 Oct 2022 at 19:54, Galen Warren <
>> > >>>> galen@cvillewarrens.com
>> > >>>> > >
>> > >>>> > >>>>>> wrote:
>> > >>>> > >>>>>>
>> > >>>> > >>>>>>> Hi Gordon (and others),
>> > >>>> > >>>>>>>
>> > >>>> > >>>>>>> I'm also using this project for stateful messaging,
>> > including
>> > >>>> > >>>>>>> messaging
>> > >>>> > >>>>>>> among functions.
>> > >>>> > >>>>>>>
>> > >>>> > >>>>>>> I've contributed a small amount of code in the past and
>> have
>> > >>>> also
>> > >>>> > >>>>>>> enabled
>> > >>>> > >>>>>>> Flink 1.15 compatibility in a local fork, so I might be
>> able
>> > >>>> to
>> > >>>> > help
>> > >>>> > >>>>>>> out
>> > >>>> > >>>>>>> here.
>> > >>>> > >>>>>>>
>> > >>>> > >>>>>>> Thanks,
>> > >>>> > >>>>>>> Galen
>> > >>>> > >>>>>>>
>> > >>>> > >>>>>>> On Wed, Oct 26, 2022 at 1:34 PM Ken Krugler <
>> > >>>> > >>>>>>> kkrugler_lists@transpac.com>
>> > >>>> > >>>>>>> wrote:
>> > >>>> > >>>>>>>
>> > >>>> > >>>>>>> > Hi Gordon,
>> > >>>> > >>>>>>> >
>> > >>>> > >>>>>>> > We’re using it for stateful messaging, and also calling
>> > >>>> remote
>> > >>>> > >>>>>>> > Python-based functions.
>> > >>>> > >>>>>>> >
>> > >>>> > >>>>>>> > So yes, also very interested in what is going to happen
>> > >>>> with the
>> > >>>> > >>>>>>> this
>> > >>>> > >>>>>>> > subproject in the future.
>> > >>>> > >>>>>>> >
>> > >>>> > >>>>>>> > — Ken
>> > >>>> > >>>>>>> >
>> > >>>> > >>>>>>> >
>> > >>>> > >>>>>>> >
>> > >>>> > >>>>>>> > > Begin forwarded message:
>> > >>>> > >>>>>>> > >
>> > >>>> > >>>>>>> > > From: "Tzu-Li (Gordon) Tai" <tz...@apache.org>
>> > >>>> > >>>>>>> > > Subject: Re: Stateful Functions with Flink 1.15 and
>> > >>>> onwards
>> > >>>> > >>>>>>> > > Date: October 26, 2022 at 10:25:26 AM PDT
>> > >>>> > >>>>>>> > > To: dev@flink.apache.org
>> > >>>> > >>>>>>> > > Reply-To: dev@flink.apache.org
>> > >>>> > >>>>>>> > >
>> > >>>> > >>>>>>> > > Hi Filip,
>> > >>>> > >>>>>>> > >
>> > >>>> > >>>>>>> > > Thanks for bringing this up.
>> > >>>> > >>>>>>> > >
>> > >>>> > >>>>>>> > > The hard truth is that committers who were previously
>> > >>>> active on
>> > >>>> > >>>>>>> the
>> > >>>> > >>>>>>> > > StateFun subproject, including myself, all currently
>> > have
>> > >>>> other
>> > >>>> > >>>>>>> focuses.
>> > >>>> > >>>>>>> > > Indeed, we may need to discuss with the community on
>> how
>> > >>>> to
>> > >>>> > >>>>>>> proceed if
>> > >>>> > >>>>>>> > > there seems to be no continued committer coverage.
>> > >>>> > >>>>>>> > >
>> > >>>> > >>>>>>> > > If it's just a matter of upgrading the supported
>> Flink
>> > >>>> version,
>> > >>>> > >>>>>>> I'm still
>> > >>>> > >>>>>>> > > familiar enough with the subproject to probably be
>> able
>> > to
>> > >>>> > drive
>> > >>>> > >>>>>>> this (or
>> > >>>> > >>>>>>> > > if your team is up to it, I can assist you on that).
>> > >>>> > >>>>>>> > >
>> > >>>> > >>>>>>> > > For the long-term, as a data point I'm curious to see
>> > how
>> > >>>> many
>> > >>>> > >>>>>>> users are
>> > >>>> > >>>>>>> > > using StateFun in production today, and how you're
>> using
>> > >>>> it?
>> > >>>> > >>>>>>> > >
>> > >>>> > >>>>>>> > > - Do your applications have arbitrary / cyclic /
>> > >>>> > bi-directional
>> > >>>> > >>>>>>> > > messaging between individual functions?
>> > >>>> > >>>>>>> > > - Or are you utilizing StateFun simply to allow
>> your
>> > >>>> stateful
>> > >>>> > >>>>>>> functions
>> > >>>> > >>>>>>> > > to run remotely as separate processes?
>> > >>>> > >>>>>>> > >
>> > >>>> > >>>>>>> > > If the majority is only the latter category, there
>> might
>> > >>>> be a
>> > >>>> > >>>>>>> case to
>> > >>>> > >>>>>>> > > support remote functions natively in Flink (which has
>> > >>>> been a
>> > >>>> > >>>>>>> discussion
>> > >>>> > >>>>>>> > in
>> > >>>> > >>>>>>> > > the past).
>> > >>>> > >>>>>>> > >
>> > >>>> > >>>>>>> > > Thanks,
>> > >>>> > >>>>>>> > > Gordon
>> > >>>> > >>>>>>> > >
>> > >>>> > >>>>>>> > > On Wed, Oct 26, 2022 at 3:30 AM Filip Karnicki <
>> > >>>> > >>>>>>> filip.karnicki@gmail.com
>> > >>>> > >>>>>>> > >
>> > >>>> > >>>>>>> > > wrote:
>> > >>>> > >>>>>>> > >
>> > >>>> > >>>>>>> > >> Hi, I noticed that the development on stateful
>> > functions
>> > >>>> has
>> > >>>> > >>>>>>> come to a
>> > >>>> > >>>>>>> > bit
>> > >>>> > >>>>>>> > >> of a halt, with a pull request to update statefun to
>> > use
>> > >>>> Flink
>> > >>>> > >>>>>>> 1.15
>> > >>>> > >>>>>>> > being
>> > >>>> > >>>>>>> > >> in the `open` state since May 2022.
>> > >>>> > >>>>>>> > >>
>> > >>>> > >>>>>>> > >> What do we think is the future of this sub-project?
>> > >>>> > >>>>>>> > >>
>> > >>>> > >>>>>>> > >> The background to this question is that my team is
>> on a
>> > >>>> shared
>> > >>>> > >>>>>>> Flink
>> > >>>> > >>>>>>> > >> cluster which will soon be upgrading to Flink 1.15.
>> If
>> > I
>> > >>>> need
>> > >>>> > to
>> > >>>> > >>>>>>> > re-write
>> > >>>> > >>>>>>> > >> all our code as a native Flink job (rather than a
>> > remote
>> > >>>> > >>>>>>> stateful
>> > >>>> > >>>>>>> > function)
>> > >>>> > >>>>>>> > >> then I need to get started right away.
>> > >>>> > >>>>>>> > >>
>> > >>>> > >>>>>>> > >> Many thanks,
>> > >>>> > >>>>>>> > >> Fil
>> > >>>> > >>>>>>> > >>
>> > >>>> > >>>>>>> >
>> > >>>> > >>>>>>> > --------------------------
>> > >>>> > >>>>>>> > Ken Krugler
>> > >>>> > >>>>>>> > http://www.scaleunlimited.com
>> > >>>> > >>>>>>> > Custom big data solutions
>> > >>>> > >>>>>>> > Flink, Pinot, Solr, Elasticsearch
>> > >>>> > >>>>>>> >
>> > >>>> > >>>>>>> >
>> > >>>> > >>>>>>> >
>> > >>>> > >>>>>>> >
>> > >>>> > >>>>>>>
>> > >>>> > >>>>>>
>> > >>>> >
>> > >>>>
>> > >>>
>> >
>>
>
Re: Stateful Functions with Flink 1.15 and onwards
Posted by Galen Warren <ga...@cvillewarrens.com>.
Hi Gordon, just checking in. Is there anything I can do to help here?
On Fri, Nov 4, 2022 at 12:59 PM Tzu-Li (Gordon) Tai <tz...@apache.org>
wrote:
> @Galen The next step is essentially for someone to be the release manager
> and drive the release process for StateFun 3.3.0 [1]. The document for the
> release process might be slightly outdated in some places, but overall
> outlines the process pretty clearly.
>
> As I mentioned earlier, there's quite a few steps in the process that
> requires committer write access, so it would be easier if a committer can
> pick this up as the release manager.
> I'll be happy to take this on, but I'll only have availability after next
> week. If someone else is willing to take this on earlier it would certainly
> be better to get the release out ASAP.
>
> [1]
>
> https://cwiki.apache.org/confluence/display/FLINK/Creating+a+Flink+Stateful+Functions+Release
>
> On Fri, Nov 4, 2022 at 5:54 AM Galen Warren <ga...@cvillewarrens.com>
> wrote:
>
> > Thanks Gordon.
> >
> > What is the next step here?
> >
> > On Thu, Nov 3, 2022 at 1:45 PM Tzu-Li (Gordon) Tai <tz...@apache.org>
> > wrote:
> >
> > > FYI, release-3.3 branch has been cut and is ready for the release
> process
> > > for StateFun 3.3.0:
> > > https://github.com/apache/flink-statefun/tree/release-3.3
> > >
> > > On Tue, Nov 1, 2022 at 10:21 AM Tzu-Li (Gordon) Tai <
> tzulitai@apache.org
> > >
> > > wrote:
> > >
> > >> Btw, I'll assume that we're using this thread to gather consensus for
> > >> code-freezing for 3.3.x series of StateFun. I know there hasn't been
> > much
> > >> activity on the repo, so this is just a formality really :)
> > >>
> > >> From the commit history, it looks like we're mainly including the
> below
> > >> major changes and bug fixes for 3.3.x:
> > >> - Flink upgrade to 1.15.2
> > >> - https://issues.apache.org/jira/browse/FLINK-26340
> > >> - https://issues.apache.org/jira/browse/FLINK-25866
> > >> - https://issues.apache.org/jira/browse/FLINK-25936
> > >> - https://issues.apache.org/jira/browse/FLINK-25933
> > >>
> > >> I'll wait for 24 hours before cutting the release branch for 3.3.x,
> > >> unless anyone raises any objections before that.
> > >>
> > >> Thanks,
> > >> Gordon
> > >>
> > >> On Tue, Nov 1, 2022 at 10:09 AM Galen Warren <galen@cvillewarrens.com
> >
> > >> wrote:
> > >>
> > >>> Thanks Gordon and Filip, I appreciate your help on this one.
> > >>>
> > >>> On Tue, Nov 1, 2022 at 1:07 PM Tzu-Li (Gordon) Tai <
> > tzulitai@apache.org>
> > >>> wrote:
> > >>>
> > >>>> PR for upgrading to Flink 1.15.2 has been merged. Thanks for the
> > >>>> efforts,
> > >>>> Galen and Filip!
> > >>>>
> > >>>> We should be ready to kick off a new release for StateFun with the
> > Flink
> > >>>> version upgrade.
> > >>>> I'll cut off a release branch now on apache/flink-statefun for
> > >>>> release-3.3.x to move things forward.
> > >>>> @Galen, @Filip if you want to, after the release branch is cut, you
> > >>>> could
> > >>>> probably upgrade the master branch to Flink 1.16.x as well.
> > >>>>
> > >>>> Afterwards we should decide who is available to drive the actual
> > release
> > >>>> process for 3.3.0.
> > >>>> There's quite a few steps that would require committer write access.
> > >>>> Unless someone else is up for this earlier, I'll have some
> > availability
> > >>>> towards the end of next week to help drive this.
> > >>>>
> > >>>> Thanks,
> > >>>> Gordon
> > >>>>
> > >>>> On Mon, Oct 31, 2022 at 12:17 PM Galen Warren <
> > galen@cvillewarrens.com>
> > >>>> wrote:
> > >>>>
> > >>>> > Yes, that makes sense.
> > >>>> >
> > >>>> > PR is here: [FLINK-29814][statefun] Change supported Flink version
> > to
> > >>>> > 1.15.2 by galenwarren · Pull Request #319 · apache/flink-statefun
> > >>>> > (github.com) <https://github.com/apache/flink-statefun/pull/319>.
> > >>>> >
> > >>>> > On Mon, Oct 31, 2022 at 11:35 AM Till Rohrmann <
> > trohrmann@apache.org>
> > >>>> > wrote:
> > >>>> >
> > >>>> > > I think there might still be value in supporting 1.15 since not
> > >>>> everyone
> > >>>> > > upgrades Flink very fast. Hopefully, for Statefun the diff
> between
> > >>>> Flink
> > >>>> > > 1.15 and 1.16 boils down to changing the Flink dependencies.
> > >>>> > >
> > >>>> > > Cheers,
> > >>>> > > Till
> > >>>> > >
> > >>>> > > On Mon, Oct 31, 2022 at 2:06 PM Galen Warren <
> > >>>> galen@cvillewarrens.com>
> > >>>> > > wrote:
> > >>>> > >
> > >>>> > >> Sure thing. One question -- Flink 1.16 was just released a few
> > >>>> days ago.
> > >>>> > >> Should I support 1.15, or just go straight to 1.16?
> > >>>> > >>
> > >>>> > >> On Mon, Oct 31, 2022 at 8:49 AM Till Rohrmann <
> > >>>> trohrmann@apache.org>
> > >>>> > >> wrote:
> > >>>> > >>
> > >>>> > >>> Hi folks,
> > >>>> > >>>
> > >>>> > >>> if you can open a PR for supporting Flink 1.15 Galen, then
> this
> > >>>> would
> > >>>> > be
> > >>>> > >>> awesome. I've assigned you to this ticket. The next thing
> after
> > >>>> merging
> > >>>> > >>> this PR would be creating a new StateFun release. Once we have
> > >>>> merged
> > >>>> > the
> > >>>> > >>> PR, let's check who can help with it the fastest.
> > >>>> > >>>
> > >>>> > >>> Cheers,
> > >>>> > >>> Till
> > >>>> > >>>
> > >>>> > >>> On Mon, Oct 31, 2022 at 1:10 PM Galen Warren <
> > >>>> galen@cvillewarrens.com>
> > >>>> > >>> wrote:
> > >>>> > >>>
> > >>>> > >>>> Yes, I could do that.
> > >>>> > >>>>
> > >>>> > >>>> On Mon, Oct 31, 2022 at 7:48 AM Filip Karnicki <
> > >>>> > >>>> filip.karnicki@gmail.com> wrote:
> > >>>> > >>>>
> > >>>> > >>>>> Hi All
> > >>>> > >>>>>
> > >>>> > >>>>> So what's the play here?
> > >>>> > >>>>>
> > >>>> > >>>>> Galen, what do you think about taking this on? Perhaps
> ++Till
> > >>>> would
> > >>>> > >>>>> assign this jira to you (with your permission) given he's
> > >>>> helped me
> > >>>> > out
> > >>>> > >>>>> with statefun work before
> > >>>> > >>>>> https://issues.apache.org/jira/browse/FLINK-29814
> > >>>> > >>>>>
> > >>>> > >>>>> I can try to move to move statefun to flink 1.16 when it's
> out
> > >>>> > >>>>>
> > >>>> > >>>>>
> > >>>> > >>>>> Kind regards
> > >>>> > >>>>> Fil
> > >>>> > >>>>>
> > >>>> > >>>>> On Thu, 27 Oct 2022 at 10:02, Filip Karnicki <
> > >>>> > filip.karnicki@gmail.com>
> > >>>> > >>>>> wrote:
> > >>>> > >>>>>
> > >>>> > >>>>>> Hi All
> > >>>> > >>>>>>
> > >>>> > >>>>>> Our use case is that we need to process elements for the
> same
> > >>>> key
> > >>>> > >>>>>> sequentially, and this processing involves async
> operations.
> > >>>> > >>>>>>
> > >>>> > >>>>>> If any part of the processing fails, we store the offending
> > >>>> and all
> > >>>> > >>>>>> subsequent incoming messages for that key in the state and
> > not
> > >>>> > process any
> > >>>> > >>>>>> further messages for that key, until a retry succeeds or a
> > >>>> human
> > >>>> > sends a
> > >>>> > >>>>>> 'skip' command message.
> > >>>> > >>>>>>
> > >>>> > >>>>>> diagram:
> > >>>> > >>>>>>
> > >>>> >
> > >>>>
> >
> https://mermaid.live/edit#pako:eNplkL1uwzAMhF-F0JQADrp76FR06tSOcQfWom3V-nFFqoUR591L20mXaqAOxHd3AC-mTZZMbfqM0wAvr00EfS62KbjYn-8C6Jui8DucTo_wmT4Oz97FEVhQqCtxXR13q_IBo-XzXWyehUc3LSu2Uyq2qIXpq2iyQ-9nmCjDSPMCmUISOuwfaEErLsVbw2272VOEDp0vmSqw5HEmC4GYsSeQpKjkv7j_buQ5tjAV4YeehOHHyQDsLAF1HbXCCyQZKB-2CTyzUOCjqUygHNBZPdxljW2MAoEaU6u0mMfGNPGqHBZJb1piasmFKlMmqxd7cqj3Dqbu0DNdfwHTGoek
> > >>>> > >>>>>> mermaid (in case mermaid.live goes down in the future):
> > >>>> > >>>>>> graph LR
> > >>>> > >>>>>> incoming[incoming events] --> job(Flink statefun job)
> > >>>> > >>>>>> commands[commands] -->|skip| job
> > >>>> > >>>>>> job --> |sequentially per key| remote(remote function)
> > >>>> > >>>>>> remote --> |on failure, delayed message to retry|
> remote
> > >>>> > >>>>>> remote --> |async puts/gets with side effects|
> > other(other
> > >>>> > >>>>>> systems)
> > >>>> > >>>>>>
> > >>>> > >>>>>> Having the processing happen outside of Flink is
> nice-to-have
> > >>>> from
> > >>>> > an
> > >>>> > >>>>>> independent scalability point of view, but is not strictly
> > >>>> required.
> > >>>> > >>>>>>
> > >>>> > >>>>>> So long story short - no cyclic messaging, but also no way
> I
> > >>>> can
> > >>>> > >>>>>> think of to use existing native Flink operators like async
> > i/o
> > >>>> > (which when
> > >>>> > >>>>>> I last checked a few years back didn't have access to keyed
> > >>>> state)
> > >>>> > >>>>>>
> > >>>> > >>>>>>
> > >>>> > >>>>>> P.S. Please note that there is already a pull request that
> > has
> > >>>> > >>>>>> something to do wtih Flink 1.15, albeit without a
> description
> > >>>> or a
> > >>>> > jira:
> > >>>> > >>>>>> https://github.com/apache/flink-statefun/pull/314
> > >>>> > >>>>>>
> > >>>> > >>>>>>
> > >>>> > >>>>>> On Wed, 26 Oct 2022 at 19:54, Galen Warren <
> > >>>> galen@cvillewarrens.com
> > >>>> > >
> > >>>> > >>>>>> wrote:
> > >>>> > >>>>>>
> > >>>> > >>>>>>> Hi Gordon (and others),
> > >>>> > >>>>>>>
> > >>>> > >>>>>>> I'm also using this project for stateful messaging,
> > including
> > >>>> > >>>>>>> messaging
> > >>>> > >>>>>>> among functions.
> > >>>> > >>>>>>>
> > >>>> > >>>>>>> I've contributed a small amount of code in the past and
> have
> > >>>> also
> > >>>> > >>>>>>> enabled
> > >>>> > >>>>>>> Flink 1.15 compatibility in a local fork, so I might be
> able
> > >>>> to
> > >>>> > help
> > >>>> > >>>>>>> out
> > >>>> > >>>>>>> here.
> > >>>> > >>>>>>>
> > >>>> > >>>>>>> Thanks,
> > >>>> > >>>>>>> Galen
> > >>>> > >>>>>>>
> > >>>> > >>>>>>> On Wed, Oct 26, 2022 at 1:34 PM Ken Krugler <
> > >>>> > >>>>>>> kkrugler_lists@transpac.com>
> > >>>> > >>>>>>> wrote:
> > >>>> > >>>>>>>
> > >>>> > >>>>>>> > Hi Gordon,
> > >>>> > >>>>>>> >
> > >>>> > >>>>>>> > We’re using it for stateful messaging, and also calling
> > >>>> remote
> > >>>> > >>>>>>> > Python-based functions.
> > >>>> > >>>>>>> >
> > >>>> > >>>>>>> > So yes, also very interested in what is going to happen
> > >>>> with the
> > >>>> > >>>>>>> this
> > >>>> > >>>>>>> > subproject in the future.
> > >>>> > >>>>>>> >
> > >>>> > >>>>>>> > — Ken
> > >>>> > >>>>>>> >
> > >>>> > >>>>>>> >
> > >>>> > >>>>>>> >
> > >>>> > >>>>>>> > > Begin forwarded message:
> > >>>> > >>>>>>> > >
> > >>>> > >>>>>>> > > From: "Tzu-Li (Gordon) Tai" <tz...@apache.org>
> > >>>> > >>>>>>> > > Subject: Re: Stateful Functions with Flink 1.15 and
> > >>>> onwards
> > >>>> > >>>>>>> > > Date: October 26, 2022 at 10:25:26 AM PDT
> > >>>> > >>>>>>> > > To: dev@flink.apache.org
> > >>>> > >>>>>>> > > Reply-To: dev@flink.apache.org
> > >>>> > >>>>>>> > >
> > >>>> > >>>>>>> > > Hi Filip,
> > >>>> > >>>>>>> > >
> > >>>> > >>>>>>> > > Thanks for bringing this up.
> > >>>> > >>>>>>> > >
> > >>>> > >>>>>>> > > The hard truth is that committers who were previously
> > >>>> active on
> > >>>> > >>>>>>> the
> > >>>> > >>>>>>> > > StateFun subproject, including myself, all currently
> > have
> > >>>> other
> > >>>> > >>>>>>> focuses.
> > >>>> > >>>>>>> > > Indeed, we may need to discuss with the community on
> how
> > >>>> to
> > >>>> > >>>>>>> proceed if
> > >>>> > >>>>>>> > > there seems to be no continued committer coverage.
> > >>>> > >>>>>>> > >
> > >>>> > >>>>>>> > > If it's just a matter of upgrading the supported Flink
> > >>>> version,
> > >>>> > >>>>>>> I'm still
> > >>>> > >>>>>>> > > familiar enough with the subproject to probably be
> able
> > to
> > >>>> > drive
> > >>>> > >>>>>>> this (or
> > >>>> > >>>>>>> > > if your team is up to it, I can assist you on that).
> > >>>> > >>>>>>> > >
> > >>>> > >>>>>>> > > For the long-term, as a data point I'm curious to see
> > how
> > >>>> many
> > >>>> > >>>>>>> users are
> > >>>> > >>>>>>> > > using StateFun in production today, and how you're
> using
> > >>>> it?
> > >>>> > >>>>>>> > >
> > >>>> > >>>>>>> > > - Do your applications have arbitrary / cyclic /
> > >>>> > bi-directional
> > >>>> > >>>>>>> > > messaging between individual functions?
> > >>>> > >>>>>>> > > - Or are you utilizing StateFun simply to allow your
> > >>>> stateful
> > >>>> > >>>>>>> functions
> > >>>> > >>>>>>> > > to run remotely as separate processes?
> > >>>> > >>>>>>> > >
> > >>>> > >>>>>>> > > If the majority is only the latter category, there
> might
> > >>>> be a
> > >>>> > >>>>>>> case to
> > >>>> > >>>>>>> > > support remote functions natively in Flink (which has
> > >>>> been a
> > >>>> > >>>>>>> discussion
> > >>>> > >>>>>>> > in
> > >>>> > >>>>>>> > > the past).
> > >>>> > >>>>>>> > >
> > >>>> > >>>>>>> > > Thanks,
> > >>>> > >>>>>>> > > Gordon
> > >>>> > >>>>>>> > >
> > >>>> > >>>>>>> > > On Wed, Oct 26, 2022 at 3:30 AM Filip Karnicki <
> > >>>> > >>>>>>> filip.karnicki@gmail.com
> > >>>> > >>>>>>> > >
> > >>>> > >>>>>>> > > wrote:
> > >>>> > >>>>>>> > >
> > >>>> > >>>>>>> > >> Hi, I noticed that the development on stateful
> > functions
> > >>>> has
> > >>>> > >>>>>>> come to a
> > >>>> > >>>>>>> > bit
> > >>>> > >>>>>>> > >> of a halt, with a pull request to update statefun to
> > use
> > >>>> Flink
> > >>>> > >>>>>>> 1.15
> > >>>> > >>>>>>> > being
> > >>>> > >>>>>>> > >> in the `open` state since May 2022.
> > >>>> > >>>>>>> > >>
> > >>>> > >>>>>>> > >> What do we think is the future of this sub-project?
> > >>>> > >>>>>>> > >>
> > >>>> > >>>>>>> > >> The background to this question is that my team is
> on a
> > >>>> shared
> > >>>> > >>>>>>> Flink
> > >>>> > >>>>>>> > >> cluster which will soon be upgrading to Flink 1.15.
> If
> > I
> > >>>> need
> > >>>> > to
> > >>>> > >>>>>>> > re-write
> > >>>> > >>>>>>> > >> all our code as a native Flink job (rather than a
> > remote
> > >>>> > >>>>>>> stateful
> > >>>> > >>>>>>> > function)
> > >>>> > >>>>>>> > >> then I need to get started right away.
> > >>>> > >>>>>>> > >>
> > >>>> > >>>>>>> > >> Many thanks,
> > >>>> > >>>>>>> > >> Fil
> > >>>> > >>>>>>> > >>
> > >>>> > >>>>>>> >
> > >>>> > >>>>>>> > --------------------------
> > >>>> > >>>>>>> > Ken Krugler
> > >>>> > >>>>>>> > http://www.scaleunlimited.com
> > >>>> > >>>>>>> > Custom big data solutions
> > >>>> > >>>>>>> > Flink, Pinot, Solr, Elasticsearch
> > >>>> > >>>>>>> >
> > >>>> > >>>>>>> >
> > >>>> > >>>>>>> >
> > >>>> > >>>>>>> >
> > >>>> > >>>>>>>
> > >>>> > >>>>>>
> > >>>> >
> > >>>>
> > >>>
> >
>
Re: Stateful Functions with Flink 1.15 and onwards
Posted by "Tzu-Li (Gordon) Tai" <tz...@apache.org>.
@Galen The next step is essentially for someone to be the release manager
and drive the release process for StateFun 3.3.0 [1]. The document for the
release process might be slightly outdated in some places, but overall
outlines the process pretty clearly.
As I mentioned earlier, there's quite a few steps in the process that
requires committer write access, so it would be easier if a committer can
pick this up as the release manager.
I'll be happy to take this on, but I'll only have availability after next
week. If someone else is willing to take this on earlier it would certainly
be better to get the release out ASAP.
[1]
https://cwiki.apache.org/confluence/display/FLINK/Creating+a+Flink+Stateful+Functions+Release
On Fri, Nov 4, 2022 at 5:54 AM Galen Warren <ga...@cvillewarrens.com> wrote:
> Thanks Gordon.
>
> What is the next step here?
>
> On Thu, Nov 3, 2022 at 1:45 PM Tzu-Li (Gordon) Tai <tz...@apache.org>
> wrote:
>
> > FYI, release-3.3 branch has been cut and is ready for the release process
> > for StateFun 3.3.0:
> > https://github.com/apache/flink-statefun/tree/release-3.3
> >
> > On Tue, Nov 1, 2022 at 10:21 AM Tzu-Li (Gordon) Tai <tzulitai@apache.org
> >
> > wrote:
> >
> >> Btw, I'll assume that we're using this thread to gather consensus for
> >> code-freezing for 3.3.x series of StateFun. I know there hasn't been
> much
> >> activity on the repo, so this is just a formality really :)
> >>
> >> From the commit history, it looks like we're mainly including the below
> >> major changes and bug fixes for 3.3.x:
> >> - Flink upgrade to 1.15.2
> >> - https://issues.apache.org/jira/browse/FLINK-26340
> >> - https://issues.apache.org/jira/browse/FLINK-25866
> >> - https://issues.apache.org/jira/browse/FLINK-25936
> >> - https://issues.apache.org/jira/browse/FLINK-25933
> >>
> >> I'll wait for 24 hours before cutting the release branch for 3.3.x,
> >> unless anyone raises any objections before that.
> >>
> >> Thanks,
> >> Gordon
> >>
> >> On Tue, Nov 1, 2022 at 10:09 AM Galen Warren <ga...@cvillewarrens.com>
> >> wrote:
> >>
> >>> Thanks Gordon and Filip, I appreciate your help on this one.
> >>>
> >>> On Tue, Nov 1, 2022 at 1:07 PM Tzu-Li (Gordon) Tai <
> tzulitai@apache.org>
> >>> wrote:
> >>>
> >>>> PR for upgrading to Flink 1.15.2 has been merged. Thanks for the
> >>>> efforts,
> >>>> Galen and Filip!
> >>>>
> >>>> We should be ready to kick off a new release for StateFun with the
> Flink
> >>>> version upgrade.
> >>>> I'll cut off a release branch now on apache/flink-statefun for
> >>>> release-3.3.x to move things forward.
> >>>> @Galen, @Filip if you want to, after the release branch is cut, you
> >>>> could
> >>>> probably upgrade the master branch to Flink 1.16.x as well.
> >>>>
> >>>> Afterwards we should decide who is available to drive the actual
> release
> >>>> process for 3.3.0.
> >>>> There's quite a few steps that would require committer write access.
> >>>> Unless someone else is up for this earlier, I'll have some
> availability
> >>>> towards the end of next week to help drive this.
> >>>>
> >>>> Thanks,
> >>>> Gordon
> >>>>
> >>>> On Mon, Oct 31, 2022 at 12:17 PM Galen Warren <
> galen@cvillewarrens.com>
> >>>> wrote:
> >>>>
> >>>> > Yes, that makes sense.
> >>>> >
> >>>> > PR is here: [FLINK-29814][statefun] Change supported Flink version
> to
> >>>> > 1.15.2 by galenwarren · Pull Request #319 · apache/flink-statefun
> >>>> > (github.com) <https://github.com/apache/flink-statefun/pull/319>.
> >>>> >
> >>>> > On Mon, Oct 31, 2022 at 11:35 AM Till Rohrmann <
> trohrmann@apache.org>
> >>>> > wrote:
> >>>> >
> >>>> > > I think there might still be value in supporting 1.15 since not
> >>>> everyone
> >>>> > > upgrades Flink very fast. Hopefully, for Statefun the diff between
> >>>> Flink
> >>>> > > 1.15 and 1.16 boils down to changing the Flink dependencies.
> >>>> > >
> >>>> > > Cheers,
> >>>> > > Till
> >>>> > >
> >>>> > > On Mon, Oct 31, 2022 at 2:06 PM Galen Warren <
> >>>> galen@cvillewarrens.com>
> >>>> > > wrote:
> >>>> > >
> >>>> > >> Sure thing. One question -- Flink 1.16 was just released a few
> >>>> days ago.
> >>>> > >> Should I support 1.15, or just go straight to 1.16?
> >>>> > >>
> >>>> > >> On Mon, Oct 31, 2022 at 8:49 AM Till Rohrmann <
> >>>> trohrmann@apache.org>
> >>>> > >> wrote:
> >>>> > >>
> >>>> > >>> Hi folks,
> >>>> > >>>
> >>>> > >>> if you can open a PR for supporting Flink 1.15 Galen, then this
> >>>> would
> >>>> > be
> >>>> > >>> awesome. I've assigned you to this ticket. The next thing after
> >>>> merging
> >>>> > >>> this PR would be creating a new StateFun release. Once we have
> >>>> merged
> >>>> > the
> >>>> > >>> PR, let's check who can help with it the fastest.
> >>>> > >>>
> >>>> > >>> Cheers,
> >>>> > >>> Till
> >>>> > >>>
> >>>> > >>> On Mon, Oct 31, 2022 at 1:10 PM Galen Warren <
> >>>> galen@cvillewarrens.com>
> >>>> > >>> wrote:
> >>>> > >>>
> >>>> > >>>> Yes, I could do that.
> >>>> > >>>>
> >>>> > >>>> On Mon, Oct 31, 2022 at 7:48 AM Filip Karnicki <
> >>>> > >>>> filip.karnicki@gmail.com> wrote:
> >>>> > >>>>
> >>>> > >>>>> Hi All
> >>>> > >>>>>
> >>>> > >>>>> So what's the play here?
> >>>> > >>>>>
> >>>> > >>>>> Galen, what do you think about taking this on? Perhaps ++Till
> >>>> would
> >>>> > >>>>> assign this jira to you (with your permission) given he's
> >>>> helped me
> >>>> > out
> >>>> > >>>>> with statefun work before
> >>>> > >>>>> https://issues.apache.org/jira/browse/FLINK-29814
> >>>> > >>>>>
> >>>> > >>>>> I can try to move to move statefun to flink 1.16 when it's out
> >>>> > >>>>>
> >>>> > >>>>>
> >>>> > >>>>> Kind regards
> >>>> > >>>>> Fil
> >>>> > >>>>>
> >>>> > >>>>> On Thu, 27 Oct 2022 at 10:02, Filip Karnicki <
> >>>> > filip.karnicki@gmail.com>
> >>>> > >>>>> wrote:
> >>>> > >>>>>
> >>>> > >>>>>> Hi All
> >>>> > >>>>>>
> >>>> > >>>>>> Our use case is that we need to process elements for the same
> >>>> key
> >>>> > >>>>>> sequentially, and this processing involves async operations.
> >>>> > >>>>>>
> >>>> > >>>>>> If any part of the processing fails, we store the offending
> >>>> and all
> >>>> > >>>>>> subsequent incoming messages for that key in the state and
> not
> >>>> > process any
> >>>> > >>>>>> further messages for that key, until a retry succeeds or a
> >>>> human
> >>>> > sends a
> >>>> > >>>>>> 'skip' command message.
> >>>> > >>>>>>
> >>>> > >>>>>> diagram:
> >>>> > >>>>>>
> >>>> >
> >>>>
> https://mermaid.live/edit#pako:eNplkL1uwzAMhF-F0JQADrp76FR06tSOcQfWom3V-nFFqoUR591L20mXaqAOxHd3AC-mTZZMbfqM0wAvr00EfS62KbjYn-8C6Jui8DucTo_wmT4Oz97FEVhQqCtxXR13q_IBo-XzXWyehUc3LSu2Uyq2qIXpq2iyQ-9nmCjDSPMCmUISOuwfaEErLsVbw2272VOEDp0vmSqw5HEmC4GYsSeQpKjkv7j_buQ5tjAV4YeehOHHyQDsLAF1HbXCCyQZKB-2CTyzUOCjqUygHNBZPdxljW2MAoEaU6u0mMfGNPGqHBZJb1piasmFKlMmqxd7cqj3Dqbu0DNdfwHTGoek
> >>>> > >>>>>> mermaid (in case mermaid.live goes down in the future):
> >>>> > >>>>>> graph LR
> >>>> > >>>>>> incoming[incoming events] --> job(Flink statefun job)
> >>>> > >>>>>> commands[commands] -->|skip| job
> >>>> > >>>>>> job --> |sequentially per key| remote(remote function)
> >>>> > >>>>>> remote --> |on failure, delayed message to retry| remote
> >>>> > >>>>>> remote --> |async puts/gets with side effects|
> other(other
> >>>> > >>>>>> systems)
> >>>> > >>>>>>
> >>>> > >>>>>> Having the processing happen outside of Flink is nice-to-have
> >>>> from
> >>>> > an
> >>>> > >>>>>> independent scalability point of view, but is not strictly
> >>>> required.
> >>>> > >>>>>>
> >>>> > >>>>>> So long story short - no cyclic messaging, but also no way I
> >>>> can
> >>>> > >>>>>> think of to use existing native Flink operators like async
> i/o
> >>>> > (which when
> >>>> > >>>>>> I last checked a few years back didn't have access to keyed
> >>>> state)
> >>>> > >>>>>>
> >>>> > >>>>>>
> >>>> > >>>>>> P.S. Please note that there is already a pull request that
> has
> >>>> > >>>>>> something to do wtih Flink 1.15, albeit without a description
> >>>> or a
> >>>> > jira:
> >>>> > >>>>>> https://github.com/apache/flink-statefun/pull/314
> >>>> > >>>>>>
> >>>> > >>>>>>
> >>>> > >>>>>> On Wed, 26 Oct 2022 at 19:54, Galen Warren <
> >>>> galen@cvillewarrens.com
> >>>> > >
> >>>> > >>>>>> wrote:
> >>>> > >>>>>>
> >>>> > >>>>>>> Hi Gordon (and others),
> >>>> > >>>>>>>
> >>>> > >>>>>>> I'm also using this project for stateful messaging,
> including
> >>>> > >>>>>>> messaging
> >>>> > >>>>>>> among functions.
> >>>> > >>>>>>>
> >>>> > >>>>>>> I've contributed a small amount of code in the past and have
> >>>> also
> >>>> > >>>>>>> enabled
> >>>> > >>>>>>> Flink 1.15 compatibility in a local fork, so I might be able
> >>>> to
> >>>> > help
> >>>> > >>>>>>> out
> >>>> > >>>>>>> here.
> >>>> > >>>>>>>
> >>>> > >>>>>>> Thanks,
> >>>> > >>>>>>> Galen
> >>>> > >>>>>>>
> >>>> > >>>>>>> On Wed, Oct 26, 2022 at 1:34 PM Ken Krugler <
> >>>> > >>>>>>> kkrugler_lists@transpac.com>
> >>>> > >>>>>>> wrote:
> >>>> > >>>>>>>
> >>>> > >>>>>>> > Hi Gordon,
> >>>> > >>>>>>> >
> >>>> > >>>>>>> > We’re using it for stateful messaging, and also calling
> >>>> remote
> >>>> > >>>>>>> > Python-based functions.
> >>>> > >>>>>>> >
> >>>> > >>>>>>> > So yes, also very interested in what is going to happen
> >>>> with the
> >>>> > >>>>>>> this
> >>>> > >>>>>>> > subproject in the future.
> >>>> > >>>>>>> >
> >>>> > >>>>>>> > — Ken
> >>>> > >>>>>>> >
> >>>> > >>>>>>> >
> >>>> > >>>>>>> >
> >>>> > >>>>>>> > > Begin forwarded message:
> >>>> > >>>>>>> > >
> >>>> > >>>>>>> > > From: "Tzu-Li (Gordon) Tai" <tz...@apache.org>
> >>>> > >>>>>>> > > Subject: Re: Stateful Functions with Flink 1.15 and
> >>>> onwards
> >>>> > >>>>>>> > > Date: October 26, 2022 at 10:25:26 AM PDT
> >>>> > >>>>>>> > > To: dev@flink.apache.org
> >>>> > >>>>>>> > > Reply-To: dev@flink.apache.org
> >>>> > >>>>>>> > >
> >>>> > >>>>>>> > > Hi Filip,
> >>>> > >>>>>>> > >
> >>>> > >>>>>>> > > Thanks for bringing this up.
> >>>> > >>>>>>> > >
> >>>> > >>>>>>> > > The hard truth is that committers who were previously
> >>>> active on
> >>>> > >>>>>>> the
> >>>> > >>>>>>> > > StateFun subproject, including myself, all currently
> have
> >>>> other
> >>>> > >>>>>>> focuses.
> >>>> > >>>>>>> > > Indeed, we may need to discuss with the community on how
> >>>> to
> >>>> > >>>>>>> proceed if
> >>>> > >>>>>>> > > there seems to be no continued committer coverage.
> >>>> > >>>>>>> > >
> >>>> > >>>>>>> > > If it's just a matter of upgrading the supported Flink
> >>>> version,
> >>>> > >>>>>>> I'm still
> >>>> > >>>>>>> > > familiar enough with the subproject to probably be able
> to
> >>>> > drive
> >>>> > >>>>>>> this (or
> >>>> > >>>>>>> > > if your team is up to it, I can assist you on that).
> >>>> > >>>>>>> > >
> >>>> > >>>>>>> > > For the long-term, as a data point I'm curious to see
> how
> >>>> many
> >>>> > >>>>>>> users are
> >>>> > >>>>>>> > > using StateFun in production today, and how you're using
> >>>> it?
> >>>> > >>>>>>> > >
> >>>> > >>>>>>> > > - Do your applications have arbitrary / cyclic /
> >>>> > bi-directional
> >>>> > >>>>>>> > > messaging between individual functions?
> >>>> > >>>>>>> > > - Or are you utilizing StateFun simply to allow your
> >>>> stateful
> >>>> > >>>>>>> functions
> >>>> > >>>>>>> > > to run remotely as separate processes?
> >>>> > >>>>>>> > >
> >>>> > >>>>>>> > > If the majority is only the latter category, there might
> >>>> be a
> >>>> > >>>>>>> case to
> >>>> > >>>>>>> > > support remote functions natively in Flink (which has
> >>>> been a
> >>>> > >>>>>>> discussion
> >>>> > >>>>>>> > in
> >>>> > >>>>>>> > > the past).
> >>>> > >>>>>>> > >
> >>>> > >>>>>>> > > Thanks,
> >>>> > >>>>>>> > > Gordon
> >>>> > >>>>>>> > >
> >>>> > >>>>>>> > > On Wed, Oct 26, 2022 at 3:30 AM Filip Karnicki <
> >>>> > >>>>>>> filip.karnicki@gmail.com
> >>>> > >>>>>>> > >
> >>>> > >>>>>>> > > wrote:
> >>>> > >>>>>>> > >
> >>>> > >>>>>>> > >> Hi, I noticed that the development on stateful
> functions
> >>>> has
> >>>> > >>>>>>> come to a
> >>>> > >>>>>>> > bit
> >>>> > >>>>>>> > >> of a halt, with a pull request to update statefun to
> use
> >>>> Flink
> >>>> > >>>>>>> 1.15
> >>>> > >>>>>>> > being
> >>>> > >>>>>>> > >> in the `open` state since May 2022.
> >>>> > >>>>>>> > >>
> >>>> > >>>>>>> > >> What do we think is the future of this sub-project?
> >>>> > >>>>>>> > >>
> >>>> > >>>>>>> > >> The background to this question is that my team is on a
> >>>> shared
> >>>> > >>>>>>> Flink
> >>>> > >>>>>>> > >> cluster which will soon be upgrading to Flink 1.15. If
> I
> >>>> need
> >>>> > to
> >>>> > >>>>>>> > re-write
> >>>> > >>>>>>> > >> all our code as a native Flink job (rather than a
> remote
> >>>> > >>>>>>> stateful
> >>>> > >>>>>>> > function)
> >>>> > >>>>>>> > >> then I need to get started right away.
> >>>> > >>>>>>> > >>
> >>>> > >>>>>>> > >> Many thanks,
> >>>> > >>>>>>> > >> Fil
> >>>> > >>>>>>> > >>
> >>>> > >>>>>>> >
> >>>> > >>>>>>> > --------------------------
> >>>> > >>>>>>> > Ken Krugler
> >>>> > >>>>>>> > http://www.scaleunlimited.com
> >>>> > >>>>>>> > Custom big data solutions
> >>>> > >>>>>>> > Flink, Pinot, Solr, Elasticsearch
> >>>> > >>>>>>> >
> >>>> > >>>>>>> >
> >>>> > >>>>>>> >
> >>>> > >>>>>>> >
> >>>> > >>>>>>>
> >>>> > >>>>>>
> >>>> >
> >>>>
> >>>
>
Re: Stateful Functions with Flink 1.15 and onwards
Posted by Galen Warren <ga...@cvillewarrens.com>.
Thanks Gordon.
What is the next step here?
On Thu, Nov 3, 2022 at 1:45 PM Tzu-Li (Gordon) Tai <tz...@apache.org>
wrote:
> FYI, release-3.3 branch has been cut and is ready for the release process
> for StateFun 3.3.0:
> https://github.com/apache/flink-statefun/tree/release-3.3
>
> On Tue, Nov 1, 2022 at 10:21 AM Tzu-Li (Gordon) Tai <tz...@apache.org>
> wrote:
>
>> Btw, I'll assume that we're using this thread to gather consensus for
>> code-freezing for 3.3.x series of StateFun. I know there hasn't been much
>> activity on the repo, so this is just a formality really :)
>>
>> From the commit history, it looks like we're mainly including the below
>> major changes and bug fixes for 3.3.x:
>> - Flink upgrade to 1.15.2
>> - https://issues.apache.org/jira/browse/FLINK-26340
>> - https://issues.apache.org/jira/browse/FLINK-25866
>> - https://issues.apache.org/jira/browse/FLINK-25936
>> - https://issues.apache.org/jira/browse/FLINK-25933
>>
>> I'll wait for 24 hours before cutting the release branch for 3.3.x,
>> unless anyone raises any objections before that.
>>
>> Thanks,
>> Gordon
>>
>> On Tue, Nov 1, 2022 at 10:09 AM Galen Warren <ga...@cvillewarrens.com>
>> wrote:
>>
>>> Thanks Gordon and Filip, I appreciate your help on this one.
>>>
>>> On Tue, Nov 1, 2022 at 1:07 PM Tzu-Li (Gordon) Tai <tz...@apache.org>
>>> wrote:
>>>
>>>> PR for upgrading to Flink 1.15.2 has been merged. Thanks for the
>>>> efforts,
>>>> Galen and Filip!
>>>>
>>>> We should be ready to kick off a new release for StateFun with the Flink
>>>> version upgrade.
>>>> I'll cut off a release branch now on apache/flink-statefun for
>>>> release-3.3.x to move things forward.
>>>> @Galen, @Filip if you want to, after the release branch is cut, you
>>>> could
>>>> probably upgrade the master branch to Flink 1.16.x as well.
>>>>
>>>> Afterwards we should decide who is available to drive the actual release
>>>> process for 3.3.0.
>>>> There's quite a few steps that would require committer write access.
>>>> Unless someone else is up for this earlier, I'll have some availability
>>>> towards the end of next week to help drive this.
>>>>
>>>> Thanks,
>>>> Gordon
>>>>
>>>> On Mon, Oct 31, 2022 at 12:17 PM Galen Warren <ga...@cvillewarrens.com>
>>>> wrote:
>>>>
>>>> > Yes, that makes sense.
>>>> >
>>>> > PR is here: [FLINK-29814][statefun] Change supported Flink version to
>>>> > 1.15.2 by galenwarren · Pull Request #319 · apache/flink-statefun
>>>> > (github.com) <https://github.com/apache/flink-statefun/pull/319>.
>>>> >
>>>> > On Mon, Oct 31, 2022 at 11:35 AM Till Rohrmann <tr...@apache.org>
>>>> > wrote:
>>>> >
>>>> > > I think there might still be value in supporting 1.15 since not
>>>> everyone
>>>> > > upgrades Flink very fast. Hopefully, for Statefun the diff between
>>>> Flink
>>>> > > 1.15 and 1.16 boils down to changing the Flink dependencies.
>>>> > >
>>>> > > Cheers,
>>>> > > Till
>>>> > >
>>>> > > On Mon, Oct 31, 2022 at 2:06 PM Galen Warren <
>>>> galen@cvillewarrens.com>
>>>> > > wrote:
>>>> > >
>>>> > >> Sure thing. One question -- Flink 1.16 was just released a few
>>>> days ago.
>>>> > >> Should I support 1.15, or just go straight to 1.16?
>>>> > >>
>>>> > >> On Mon, Oct 31, 2022 at 8:49 AM Till Rohrmann <
>>>> trohrmann@apache.org>
>>>> > >> wrote:
>>>> > >>
>>>> > >>> Hi folks,
>>>> > >>>
>>>> > >>> if you can open a PR for supporting Flink 1.15 Galen, then this
>>>> would
>>>> > be
>>>> > >>> awesome. I've assigned you to this ticket. The next thing after
>>>> merging
>>>> > >>> this PR would be creating a new StateFun release. Once we have
>>>> merged
>>>> > the
>>>> > >>> PR, let's check who can help with it the fastest.
>>>> > >>>
>>>> > >>> Cheers,
>>>> > >>> Till
>>>> > >>>
>>>> > >>> On Mon, Oct 31, 2022 at 1:10 PM Galen Warren <
>>>> galen@cvillewarrens.com>
>>>> > >>> wrote:
>>>> > >>>
>>>> > >>>> Yes, I could do that.
>>>> > >>>>
>>>> > >>>> On Mon, Oct 31, 2022 at 7:48 AM Filip Karnicki <
>>>> > >>>> filip.karnicki@gmail.com> wrote:
>>>> > >>>>
>>>> > >>>>> Hi All
>>>> > >>>>>
>>>> > >>>>> So what's the play here?
>>>> > >>>>>
>>>> > >>>>> Galen, what do you think about taking this on? Perhaps ++Till
>>>> would
>>>> > >>>>> assign this jira to you (with your permission) given he's
>>>> helped me
>>>> > out
>>>> > >>>>> with statefun work before
>>>> > >>>>> https://issues.apache.org/jira/browse/FLINK-29814
>>>> > >>>>>
>>>> > >>>>> I can try to move to move statefun to flink 1.16 when it's out
>>>> > >>>>>
>>>> > >>>>>
>>>> > >>>>> Kind regards
>>>> > >>>>> Fil
>>>> > >>>>>
>>>> > >>>>> On Thu, 27 Oct 2022 at 10:02, Filip Karnicki <
>>>> > filip.karnicki@gmail.com>
>>>> > >>>>> wrote:
>>>> > >>>>>
>>>> > >>>>>> Hi All
>>>> > >>>>>>
>>>> > >>>>>> Our use case is that we need to process elements for the same
>>>> key
>>>> > >>>>>> sequentially, and this processing involves async operations.
>>>> > >>>>>>
>>>> > >>>>>> If any part of the processing fails, we store the offending
>>>> and all
>>>> > >>>>>> subsequent incoming messages for that key in the state and not
>>>> > process any
>>>> > >>>>>> further messages for that key, until a retry succeeds or a
>>>> human
>>>> > sends a
>>>> > >>>>>> 'skip' command message.
>>>> > >>>>>>
>>>> > >>>>>> diagram:
>>>> > >>>>>>
>>>> >
>>>> https://mermaid.live/edit#pako:eNplkL1uwzAMhF-F0JQADrp76FR06tSOcQfWom3V-nFFqoUR591L20mXaqAOxHd3AC-mTZZMbfqM0wAvr00EfS62KbjYn-8C6Jui8DucTo_wmT4Oz97FEVhQqCtxXR13q_IBo-XzXWyehUc3LSu2Uyq2qIXpq2iyQ-9nmCjDSPMCmUISOuwfaEErLsVbw2272VOEDp0vmSqw5HEmC4GYsSeQpKjkv7j_buQ5tjAV4YeehOHHyQDsLAF1HbXCCyQZKB-2CTyzUOCjqUygHNBZPdxljW2MAoEaU6u0mMfGNPGqHBZJb1piasmFKlMmqxd7cqj3Dqbu0DNdfwHTGoek
>>>> > >>>>>> mermaid (in case mermaid.live goes down in the future):
>>>> > >>>>>> graph LR
>>>> > >>>>>> incoming[incoming events] --> job(Flink statefun job)
>>>> > >>>>>> commands[commands] -->|skip| job
>>>> > >>>>>> job --> |sequentially per key| remote(remote function)
>>>> > >>>>>> remote --> |on failure, delayed message to retry| remote
>>>> > >>>>>> remote --> |async puts/gets with side effects| other(other
>>>> > >>>>>> systems)
>>>> > >>>>>>
>>>> > >>>>>> Having the processing happen outside of Flink is nice-to-have
>>>> from
>>>> > an
>>>> > >>>>>> independent scalability point of view, but is not strictly
>>>> required.
>>>> > >>>>>>
>>>> > >>>>>> So long story short - no cyclic messaging, but also no way I
>>>> can
>>>> > >>>>>> think of to use existing native Flink operators like async i/o
>>>> > (which when
>>>> > >>>>>> I last checked a few years back didn't have access to keyed
>>>> state)
>>>> > >>>>>>
>>>> > >>>>>>
>>>> > >>>>>> P.S. Please note that there is already a pull request that has
>>>> > >>>>>> something to do wtih Flink 1.15, albeit without a description
>>>> or a
>>>> > jira:
>>>> > >>>>>> https://github.com/apache/flink-statefun/pull/314
>>>> > >>>>>>
>>>> > >>>>>>
>>>> > >>>>>> On Wed, 26 Oct 2022 at 19:54, Galen Warren <
>>>> galen@cvillewarrens.com
>>>> > >
>>>> > >>>>>> wrote:
>>>> > >>>>>>
>>>> > >>>>>>> Hi Gordon (and others),
>>>> > >>>>>>>
>>>> > >>>>>>> I'm also using this project for stateful messaging, including
>>>> > >>>>>>> messaging
>>>> > >>>>>>> among functions.
>>>> > >>>>>>>
>>>> > >>>>>>> I've contributed a small amount of code in the past and have
>>>> also
>>>> > >>>>>>> enabled
>>>> > >>>>>>> Flink 1.15 compatibility in a local fork, so I might be able
>>>> to
>>>> > help
>>>> > >>>>>>> out
>>>> > >>>>>>> here.
>>>> > >>>>>>>
>>>> > >>>>>>> Thanks,
>>>> > >>>>>>> Galen
>>>> > >>>>>>>
>>>> > >>>>>>> On Wed, Oct 26, 2022 at 1:34 PM Ken Krugler <
>>>> > >>>>>>> kkrugler_lists@transpac.com>
>>>> > >>>>>>> wrote:
>>>> > >>>>>>>
>>>> > >>>>>>> > Hi Gordon,
>>>> > >>>>>>> >
>>>> > >>>>>>> > We’re using it for stateful messaging, and also calling
>>>> remote
>>>> > >>>>>>> > Python-based functions.
>>>> > >>>>>>> >
>>>> > >>>>>>> > So yes, also very interested in what is going to happen
>>>> with the
>>>> > >>>>>>> this
>>>> > >>>>>>> > subproject in the future.
>>>> > >>>>>>> >
>>>> > >>>>>>> > — Ken
>>>> > >>>>>>> >
>>>> > >>>>>>> >
>>>> > >>>>>>> >
>>>> > >>>>>>> > > Begin forwarded message:
>>>> > >>>>>>> > >
>>>> > >>>>>>> > > From: "Tzu-Li (Gordon) Tai" <tz...@apache.org>
>>>> > >>>>>>> > > Subject: Re: Stateful Functions with Flink 1.15 and
>>>> onwards
>>>> > >>>>>>> > > Date: October 26, 2022 at 10:25:26 AM PDT
>>>> > >>>>>>> > > To: dev@flink.apache.org
>>>> > >>>>>>> > > Reply-To: dev@flink.apache.org
>>>> > >>>>>>> > >
>>>> > >>>>>>> > > Hi Filip,
>>>> > >>>>>>> > >
>>>> > >>>>>>> > > Thanks for bringing this up.
>>>> > >>>>>>> > >
>>>> > >>>>>>> > > The hard truth is that committers who were previously
>>>> active on
>>>> > >>>>>>> the
>>>> > >>>>>>> > > StateFun subproject, including myself, all currently have
>>>> other
>>>> > >>>>>>> focuses.
>>>> > >>>>>>> > > Indeed, we may need to discuss with the community on how
>>>> to
>>>> > >>>>>>> proceed if
>>>> > >>>>>>> > > there seems to be no continued committer coverage.
>>>> > >>>>>>> > >
>>>> > >>>>>>> > > If it's just a matter of upgrading the supported Flink
>>>> version,
>>>> > >>>>>>> I'm still
>>>> > >>>>>>> > > familiar enough with the subproject to probably be able to
>>>> > drive
>>>> > >>>>>>> this (or
>>>> > >>>>>>> > > if your team is up to it, I can assist you on that).
>>>> > >>>>>>> > >
>>>> > >>>>>>> > > For the long-term, as a data point I'm curious to see how
>>>> many
>>>> > >>>>>>> users are
>>>> > >>>>>>> > > using StateFun in production today, and how you're using
>>>> it?
>>>> > >>>>>>> > >
>>>> > >>>>>>> > > - Do your applications have arbitrary / cyclic /
>>>> > bi-directional
>>>> > >>>>>>> > > messaging between individual functions?
>>>> > >>>>>>> > > - Or are you utilizing StateFun simply to allow your
>>>> stateful
>>>> > >>>>>>> functions
>>>> > >>>>>>> > > to run remotely as separate processes?
>>>> > >>>>>>> > >
>>>> > >>>>>>> > > If the majority is only the latter category, there might
>>>> be a
>>>> > >>>>>>> case to
>>>> > >>>>>>> > > support remote functions natively in Flink (which has
>>>> been a
>>>> > >>>>>>> discussion
>>>> > >>>>>>> > in
>>>> > >>>>>>> > > the past).
>>>> > >>>>>>> > >
>>>> > >>>>>>> > > Thanks,
>>>> > >>>>>>> > > Gordon
>>>> > >>>>>>> > >
>>>> > >>>>>>> > > On Wed, Oct 26, 2022 at 3:30 AM Filip Karnicki <
>>>> > >>>>>>> filip.karnicki@gmail.com
>>>> > >>>>>>> > >
>>>> > >>>>>>> > > wrote:
>>>> > >>>>>>> > >
>>>> > >>>>>>> > >> Hi, I noticed that the development on stateful functions
>>>> has
>>>> > >>>>>>> come to a
>>>> > >>>>>>> > bit
>>>> > >>>>>>> > >> of a halt, with a pull request to update statefun to use
>>>> Flink
>>>> > >>>>>>> 1.15
>>>> > >>>>>>> > being
>>>> > >>>>>>> > >> in the `open` state since May 2022.
>>>> > >>>>>>> > >>
>>>> > >>>>>>> > >> What do we think is the future of this sub-project?
>>>> > >>>>>>> > >>
>>>> > >>>>>>> > >> The background to this question is that my team is on a
>>>> shared
>>>> > >>>>>>> Flink
>>>> > >>>>>>> > >> cluster which will soon be upgrading to Flink 1.15. If I
>>>> need
>>>> > to
>>>> > >>>>>>> > re-write
>>>> > >>>>>>> > >> all our code as a native Flink job (rather than a remote
>>>> > >>>>>>> stateful
>>>> > >>>>>>> > function)
>>>> > >>>>>>> > >> then I need to get started right away.
>>>> > >>>>>>> > >>
>>>> > >>>>>>> > >> Many thanks,
>>>> > >>>>>>> > >> Fil
>>>> > >>>>>>> > >>
>>>> > >>>>>>> >
>>>> > >>>>>>> > --------------------------
>>>> > >>>>>>> > Ken Krugler
>>>> > >>>>>>> > http://www.scaleunlimited.com
>>>> > >>>>>>> > Custom big data solutions
>>>> > >>>>>>> > Flink, Pinot, Solr, Elasticsearch
>>>> > >>>>>>> >
>>>> > >>>>>>> >
>>>> > >>>>>>> >
>>>> > >>>>>>> >
>>>> > >>>>>>>
>>>> > >>>>>>
>>>> >
>>>>
>>>
Re: Stateful Functions with Flink 1.15 and onwards
Posted by "Tzu-Li (Gordon) Tai" <tz...@apache.org>.
FYI, release-3.3 branch has been cut and is ready for the release process
for StateFun 3.3.0:
https://github.com/apache/flink-statefun/tree/release-3.3
On Tue, Nov 1, 2022 at 10:21 AM Tzu-Li (Gordon) Tai <tz...@apache.org>
wrote:
> Btw, I'll assume that we're using this thread to gather consensus for
> code-freezing for 3.3.x series of StateFun. I know there hasn't been much
> activity on the repo, so this is just a formality really :)
>
> From the commit history, it looks like we're mainly including the below
> major changes and bug fixes for 3.3.x:
> - Flink upgrade to 1.15.2
> - https://issues.apache.org/jira/browse/FLINK-26340
> - https://issues.apache.org/jira/browse/FLINK-25866
> - https://issues.apache.org/jira/browse/FLINK-25936
> - https://issues.apache.org/jira/browse/FLINK-25933
>
> I'll wait for 24 hours before cutting the release branch for 3.3.x, unless
> anyone raises any objections before that.
>
> Thanks,
> Gordon
>
> On Tue, Nov 1, 2022 at 10:09 AM Galen Warren <ga...@cvillewarrens.com>
> wrote:
>
>> Thanks Gordon and Filip, I appreciate your help on this one.
>>
>> On Tue, Nov 1, 2022 at 1:07 PM Tzu-Li (Gordon) Tai <tz...@apache.org>
>> wrote:
>>
>>> PR for upgrading to Flink 1.15.2 has been merged. Thanks for the efforts,
>>> Galen and Filip!
>>>
>>> We should be ready to kick off a new release for StateFun with the Flink
>>> version upgrade.
>>> I'll cut off a release branch now on apache/flink-statefun for
>>> release-3.3.x to move things forward.
>>> @Galen, @Filip if you want to, after the release branch is cut, you could
>>> probably upgrade the master branch to Flink 1.16.x as well.
>>>
>>> Afterwards we should decide who is available to drive the actual release
>>> process for 3.3.0.
>>> There's quite a few steps that would require committer write access.
>>> Unless someone else is up for this earlier, I'll have some availability
>>> towards the end of next week to help drive this.
>>>
>>> Thanks,
>>> Gordon
>>>
>>> On Mon, Oct 31, 2022 at 12:17 PM Galen Warren <ga...@cvillewarrens.com>
>>> wrote:
>>>
>>> > Yes, that makes sense.
>>> >
>>> > PR is here: [FLINK-29814][statefun] Change supported Flink version to
>>> > 1.15.2 by galenwarren · Pull Request #319 · apache/flink-statefun
>>> > (github.com) <https://github.com/apache/flink-statefun/pull/319>.
>>> >
>>> > On Mon, Oct 31, 2022 at 11:35 AM Till Rohrmann <tr...@apache.org>
>>> > wrote:
>>> >
>>> > > I think there might still be value in supporting 1.15 since not
>>> everyone
>>> > > upgrades Flink very fast. Hopefully, for Statefun the diff between
>>> Flink
>>> > > 1.15 and 1.16 boils down to changing the Flink dependencies.
>>> > >
>>> > > Cheers,
>>> > > Till
>>> > >
>>> > > On Mon, Oct 31, 2022 at 2:06 PM Galen Warren <
>>> galen@cvillewarrens.com>
>>> > > wrote:
>>> > >
>>> > >> Sure thing. One question -- Flink 1.16 was just released a few days
>>> ago.
>>> > >> Should I support 1.15, or just go straight to 1.16?
>>> > >>
>>> > >> On Mon, Oct 31, 2022 at 8:49 AM Till Rohrmann <trohrmann@apache.org
>>> >
>>> > >> wrote:
>>> > >>
>>> > >>> Hi folks,
>>> > >>>
>>> > >>> if you can open a PR for supporting Flink 1.15 Galen, then this
>>> would
>>> > be
>>> > >>> awesome. I've assigned you to this ticket. The next thing after
>>> merging
>>> > >>> this PR would be creating a new StateFun release. Once we have
>>> merged
>>> > the
>>> > >>> PR, let's check who can help with it the fastest.
>>> > >>>
>>> > >>> Cheers,
>>> > >>> Till
>>> > >>>
>>> > >>> On Mon, Oct 31, 2022 at 1:10 PM Galen Warren <
>>> galen@cvillewarrens.com>
>>> > >>> wrote:
>>> > >>>
>>> > >>>> Yes, I could do that.
>>> > >>>>
>>> > >>>> On Mon, Oct 31, 2022 at 7:48 AM Filip Karnicki <
>>> > >>>> filip.karnicki@gmail.com> wrote:
>>> > >>>>
>>> > >>>>> Hi All
>>> > >>>>>
>>> > >>>>> So what's the play here?
>>> > >>>>>
>>> > >>>>> Galen, what do you think about taking this on? Perhaps ++Till
>>> would
>>> > >>>>> assign this jira to you (with your permission) given he's helped
>>> me
>>> > out
>>> > >>>>> with statefun work before
>>> > >>>>> https://issues.apache.org/jira/browse/FLINK-29814
>>> > >>>>>
>>> > >>>>> I can try to move to move statefun to flink 1.16 when it's out
>>> > >>>>>
>>> > >>>>>
>>> > >>>>> Kind regards
>>> > >>>>> Fil
>>> > >>>>>
>>> > >>>>> On Thu, 27 Oct 2022 at 10:02, Filip Karnicki <
>>> > filip.karnicki@gmail.com>
>>> > >>>>> wrote:
>>> > >>>>>
>>> > >>>>>> Hi All
>>> > >>>>>>
>>> > >>>>>> Our use case is that we need to process elements for the same
>>> key
>>> > >>>>>> sequentially, and this processing involves async operations.
>>> > >>>>>>
>>> > >>>>>> If any part of the processing fails, we store the offending and
>>> all
>>> > >>>>>> subsequent incoming messages for that key in the state and not
>>> > process any
>>> > >>>>>> further messages for that key, until a retry succeeds or a human
>>> > sends a
>>> > >>>>>> 'skip' command message.
>>> > >>>>>>
>>> > >>>>>> diagram:
>>> > >>>>>>
>>> >
>>> https://mermaid.live/edit#pako:eNplkL1uwzAMhF-F0JQADrp76FR06tSOcQfWom3V-nFFqoUR591L20mXaqAOxHd3AC-mTZZMbfqM0wAvr00EfS62KbjYn-8C6Jui8DucTo_wmT4Oz97FEVhQqCtxXR13q_IBo-XzXWyehUc3LSu2Uyq2qIXpq2iyQ-9nmCjDSPMCmUISOuwfaEErLsVbw2272VOEDp0vmSqw5HEmC4GYsSeQpKjkv7j_buQ5tjAV4YeehOHHyQDsLAF1HbXCCyQZKB-2CTyzUOCjqUygHNBZPdxljW2MAoEaU6u0mMfGNPGqHBZJb1piasmFKlMmqxd7cqj3Dqbu0DNdfwHTGoek
>>> > >>>>>> mermaid (in case mermaid.live goes down in the future):
>>> > >>>>>> graph LR
>>> > >>>>>> incoming[incoming events] --> job(Flink statefun job)
>>> > >>>>>> commands[commands] -->|skip| job
>>> > >>>>>> job --> |sequentially per key| remote(remote function)
>>> > >>>>>> remote --> |on failure, delayed message to retry| remote
>>> > >>>>>> remote --> |async puts/gets with side effects| other(other
>>> > >>>>>> systems)
>>> > >>>>>>
>>> > >>>>>> Having the processing happen outside of Flink is nice-to-have
>>> from
>>> > an
>>> > >>>>>> independent scalability point of view, but is not strictly
>>> required.
>>> > >>>>>>
>>> > >>>>>> So long story short - no cyclic messaging, but also no way I can
>>> > >>>>>> think of to use existing native Flink operators like async i/o
>>> > (which when
>>> > >>>>>> I last checked a few years back didn't have access to keyed
>>> state)
>>> > >>>>>>
>>> > >>>>>>
>>> > >>>>>> P.S. Please note that there is already a pull request that has
>>> > >>>>>> something to do wtih Flink 1.15, albeit without a description
>>> or a
>>> > jira:
>>> > >>>>>> https://github.com/apache/flink-statefun/pull/314
>>> > >>>>>>
>>> > >>>>>>
>>> > >>>>>> On Wed, 26 Oct 2022 at 19:54, Galen Warren <
>>> galen@cvillewarrens.com
>>> > >
>>> > >>>>>> wrote:
>>> > >>>>>>
>>> > >>>>>>> Hi Gordon (and others),
>>> > >>>>>>>
>>> > >>>>>>> I'm also using this project for stateful messaging, including
>>> > >>>>>>> messaging
>>> > >>>>>>> among functions.
>>> > >>>>>>>
>>> > >>>>>>> I've contributed a small amount of code in the past and have
>>> also
>>> > >>>>>>> enabled
>>> > >>>>>>> Flink 1.15 compatibility in a local fork, so I might be able to
>>> > help
>>> > >>>>>>> out
>>> > >>>>>>> here.
>>> > >>>>>>>
>>> > >>>>>>> Thanks,
>>> > >>>>>>> Galen
>>> > >>>>>>>
>>> > >>>>>>> On Wed, Oct 26, 2022 at 1:34 PM Ken Krugler <
>>> > >>>>>>> kkrugler_lists@transpac.com>
>>> > >>>>>>> wrote:
>>> > >>>>>>>
>>> > >>>>>>> > Hi Gordon,
>>> > >>>>>>> >
>>> > >>>>>>> > We’re using it for stateful messaging, and also calling
>>> remote
>>> > >>>>>>> > Python-based functions.
>>> > >>>>>>> >
>>> > >>>>>>> > So yes, also very interested in what is going to happen with
>>> the
>>> > >>>>>>> this
>>> > >>>>>>> > subproject in the future.
>>> > >>>>>>> >
>>> > >>>>>>> > — Ken
>>> > >>>>>>> >
>>> > >>>>>>> >
>>> > >>>>>>> >
>>> > >>>>>>> > > Begin forwarded message:
>>> > >>>>>>> > >
>>> > >>>>>>> > > From: "Tzu-Li (Gordon) Tai" <tz...@apache.org>
>>> > >>>>>>> > > Subject: Re: Stateful Functions with Flink 1.15 and onwards
>>> > >>>>>>> > > Date: October 26, 2022 at 10:25:26 AM PDT
>>> > >>>>>>> > > To: dev@flink.apache.org
>>> > >>>>>>> > > Reply-To: dev@flink.apache.org
>>> > >>>>>>> > >
>>> > >>>>>>> > > Hi Filip,
>>> > >>>>>>> > >
>>> > >>>>>>> > > Thanks for bringing this up.
>>> > >>>>>>> > >
>>> > >>>>>>> > > The hard truth is that committers who were previously
>>> active on
>>> > >>>>>>> the
>>> > >>>>>>> > > StateFun subproject, including myself, all currently have
>>> other
>>> > >>>>>>> focuses.
>>> > >>>>>>> > > Indeed, we may need to discuss with the community on how to
>>> > >>>>>>> proceed if
>>> > >>>>>>> > > there seems to be no continued committer coverage.
>>> > >>>>>>> > >
>>> > >>>>>>> > > If it's just a matter of upgrading the supported Flink
>>> version,
>>> > >>>>>>> I'm still
>>> > >>>>>>> > > familiar enough with the subproject to probably be able to
>>> > drive
>>> > >>>>>>> this (or
>>> > >>>>>>> > > if your team is up to it, I can assist you on that).
>>> > >>>>>>> > >
>>> > >>>>>>> > > For the long-term, as a data point I'm curious to see how
>>> many
>>> > >>>>>>> users are
>>> > >>>>>>> > > using StateFun in production today, and how you're using
>>> it?
>>> > >>>>>>> > >
>>> > >>>>>>> > > - Do your applications have arbitrary / cyclic /
>>> > bi-directional
>>> > >>>>>>> > > messaging between individual functions?
>>> > >>>>>>> > > - Or are you utilizing StateFun simply to allow your
>>> stateful
>>> > >>>>>>> functions
>>> > >>>>>>> > > to run remotely as separate processes?
>>> > >>>>>>> > >
>>> > >>>>>>> > > If the majority is only the latter category, there might
>>> be a
>>> > >>>>>>> case to
>>> > >>>>>>> > > support remote functions natively in Flink (which has been
>>> a
>>> > >>>>>>> discussion
>>> > >>>>>>> > in
>>> > >>>>>>> > > the past).
>>> > >>>>>>> > >
>>> > >>>>>>> > > Thanks,
>>> > >>>>>>> > > Gordon
>>> > >>>>>>> > >
>>> > >>>>>>> > > On Wed, Oct 26, 2022 at 3:30 AM Filip Karnicki <
>>> > >>>>>>> filip.karnicki@gmail.com
>>> > >>>>>>> > >
>>> > >>>>>>> > > wrote:
>>> > >>>>>>> > >
>>> > >>>>>>> > >> Hi, I noticed that the development on stateful functions
>>> has
>>> > >>>>>>> come to a
>>> > >>>>>>> > bit
>>> > >>>>>>> > >> of a halt, with a pull request to update statefun to use
>>> Flink
>>> > >>>>>>> 1.15
>>> > >>>>>>> > being
>>> > >>>>>>> > >> in the `open` state since May 2022.
>>> > >>>>>>> > >>
>>> > >>>>>>> > >> What do we think is the future of this sub-project?
>>> > >>>>>>> > >>
>>> > >>>>>>> > >> The background to this question is that my team is on a
>>> shared
>>> > >>>>>>> Flink
>>> > >>>>>>> > >> cluster which will soon be upgrading to Flink 1.15. If I
>>> need
>>> > to
>>> > >>>>>>> > re-write
>>> > >>>>>>> > >> all our code as a native Flink job (rather than a remote
>>> > >>>>>>> stateful
>>> > >>>>>>> > function)
>>> > >>>>>>> > >> then I need to get started right away.
>>> > >>>>>>> > >>
>>> > >>>>>>> > >> Many thanks,
>>> > >>>>>>> > >> Fil
>>> > >>>>>>> > >>
>>> > >>>>>>> >
>>> > >>>>>>> > --------------------------
>>> > >>>>>>> > Ken Krugler
>>> > >>>>>>> > http://www.scaleunlimited.com
>>> > >>>>>>> > Custom big data solutions
>>> > >>>>>>> > Flink, Pinot, Solr, Elasticsearch
>>> > >>>>>>> >
>>> > >>>>>>> >
>>> > >>>>>>> >
>>> > >>>>>>> >
>>> > >>>>>>>
>>> > >>>>>>
>>> >
>>>
>>
Re: Stateful Functions with Flink 1.15 and onwards
Posted by "Tzu-Li (Gordon) Tai" <tz...@apache.org>.
Btw, I'll assume that we're using this thread to gather consensus for
code-freezing for 3.3.x series of StateFun. I know there hasn't been much
activity on the repo, so this is just a formality really :)
From the commit history, it looks like we're mainly including the below
major changes and bug fixes for 3.3.x:
- Flink upgrade to 1.15.2
- https://issues.apache.org/jira/browse/FLINK-26340
- https://issues.apache.org/jira/browse/FLINK-25866
- https://issues.apache.org/jira/browse/FLINK-25936
- https://issues.apache.org/jira/browse/FLINK-25933
I'll wait for 24 hours before cutting the release branch for 3.3.x, unless
anyone raises any objections before that.
Thanks,
Gordon
On Tue, Nov 1, 2022 at 10:09 AM Galen Warren <ga...@cvillewarrens.com>
wrote:
> Thanks Gordon and Filip, I appreciate your help on this one.
>
> On Tue, Nov 1, 2022 at 1:07 PM Tzu-Li (Gordon) Tai <tz...@apache.org>
> wrote:
>
>> PR for upgrading to Flink 1.15.2 has been merged. Thanks for the efforts,
>> Galen and Filip!
>>
>> We should be ready to kick off a new release for StateFun with the Flink
>> version upgrade.
>> I'll cut off a release branch now on apache/flink-statefun for
>> release-3.3.x to move things forward.
>> @Galen, @Filip if you want to, after the release branch is cut, you could
>> probably upgrade the master branch to Flink 1.16.x as well.
>>
>> Afterwards we should decide who is available to drive the actual release
>> process for 3.3.0.
>> There's quite a few steps that would require committer write access.
>> Unless someone else is up for this earlier, I'll have some availability
>> towards the end of next week to help drive this.
>>
>> Thanks,
>> Gordon
>>
>> On Mon, Oct 31, 2022 at 12:17 PM Galen Warren <ga...@cvillewarrens.com>
>> wrote:
>>
>> > Yes, that makes sense.
>> >
>> > PR is here: [FLINK-29814][statefun] Change supported Flink version to
>> > 1.15.2 by galenwarren · Pull Request #319 · apache/flink-statefun
>> > (github.com) <https://github.com/apache/flink-statefun/pull/319>.
>> >
>> > On Mon, Oct 31, 2022 at 11:35 AM Till Rohrmann <tr...@apache.org>
>> > wrote:
>> >
>> > > I think there might still be value in supporting 1.15 since not
>> everyone
>> > > upgrades Flink very fast. Hopefully, for Statefun the diff between
>> Flink
>> > > 1.15 and 1.16 boils down to changing the Flink dependencies.
>> > >
>> > > Cheers,
>> > > Till
>> > >
>> > > On Mon, Oct 31, 2022 at 2:06 PM Galen Warren <galen@cvillewarrens.com
>> >
>> > > wrote:
>> > >
>> > >> Sure thing. One question -- Flink 1.16 was just released a few days
>> ago.
>> > >> Should I support 1.15, or just go straight to 1.16?
>> > >>
>> > >> On Mon, Oct 31, 2022 at 8:49 AM Till Rohrmann <tr...@apache.org>
>> > >> wrote:
>> > >>
>> > >>> Hi folks,
>> > >>>
>> > >>> if you can open a PR for supporting Flink 1.15 Galen, then this
>> would
>> > be
>> > >>> awesome. I've assigned you to this ticket. The next thing after
>> merging
>> > >>> this PR would be creating a new StateFun release. Once we have
>> merged
>> > the
>> > >>> PR, let's check who can help with it the fastest.
>> > >>>
>> > >>> Cheers,
>> > >>> Till
>> > >>>
>> > >>> On Mon, Oct 31, 2022 at 1:10 PM Galen Warren <
>> galen@cvillewarrens.com>
>> > >>> wrote:
>> > >>>
>> > >>>> Yes, I could do that.
>> > >>>>
>> > >>>> On Mon, Oct 31, 2022 at 7:48 AM Filip Karnicki <
>> > >>>> filip.karnicki@gmail.com> wrote:
>> > >>>>
>> > >>>>> Hi All
>> > >>>>>
>> > >>>>> So what's the play here?
>> > >>>>>
>> > >>>>> Galen, what do you think about taking this on? Perhaps ++Till
>> would
>> > >>>>> assign this jira to you (with your permission) given he's helped
>> me
>> > out
>> > >>>>> with statefun work before
>> > >>>>> https://issues.apache.org/jira/browse/FLINK-29814
>> > >>>>>
>> > >>>>> I can try to move to move statefun to flink 1.16 when it's out
>> > >>>>>
>> > >>>>>
>> > >>>>> Kind regards
>> > >>>>> Fil
>> > >>>>>
>> > >>>>> On Thu, 27 Oct 2022 at 10:02, Filip Karnicki <
>> > filip.karnicki@gmail.com>
>> > >>>>> wrote:
>> > >>>>>
>> > >>>>>> Hi All
>> > >>>>>>
>> > >>>>>> Our use case is that we need to process elements for the same key
>> > >>>>>> sequentially, and this processing involves async operations.
>> > >>>>>>
>> > >>>>>> If any part of the processing fails, we store the offending and
>> all
>> > >>>>>> subsequent incoming messages for that key in the state and not
>> > process any
>> > >>>>>> further messages for that key, until a retry succeeds or a human
>> > sends a
>> > >>>>>> 'skip' command message.
>> > >>>>>>
>> > >>>>>> diagram:
>> > >>>>>>
>> >
>> https://mermaid.live/edit#pako:eNplkL1uwzAMhF-F0JQADrp76FR06tSOcQfWom3V-nFFqoUR591L20mXaqAOxHd3AC-mTZZMbfqM0wAvr00EfS62KbjYn-8C6Jui8DucTo_wmT4Oz97FEVhQqCtxXR13q_IBo-XzXWyehUc3LSu2Uyq2qIXpq2iyQ-9nmCjDSPMCmUISOuwfaEErLsVbw2272VOEDp0vmSqw5HEmC4GYsSeQpKjkv7j_buQ5tjAV4YeehOHHyQDsLAF1HbXCCyQZKB-2CTyzUOCjqUygHNBZPdxljW2MAoEaU6u0mMfGNPGqHBZJb1piasmFKlMmqxd7cqj3Dqbu0DNdfwHTGoek
>> > >>>>>> mermaid (in case mermaid.live goes down in the future):
>> > >>>>>> graph LR
>> > >>>>>> incoming[incoming events] --> job(Flink statefun job)
>> > >>>>>> commands[commands] -->|skip| job
>> > >>>>>> job --> |sequentially per key| remote(remote function)
>> > >>>>>> remote --> |on failure, delayed message to retry| remote
>> > >>>>>> remote --> |async puts/gets with side effects| other(other
>> > >>>>>> systems)
>> > >>>>>>
>> > >>>>>> Having the processing happen outside of Flink is nice-to-have
>> from
>> > an
>> > >>>>>> independent scalability point of view, but is not strictly
>> required.
>> > >>>>>>
>> > >>>>>> So long story short - no cyclic messaging, but also no way I can
>> > >>>>>> think of to use existing native Flink operators like async i/o
>> > (which when
>> > >>>>>> I last checked a few years back didn't have access to keyed
>> state)
>> > >>>>>>
>> > >>>>>>
>> > >>>>>> P.S. Please note that there is already a pull request that has
>> > >>>>>> something to do wtih Flink 1.15, albeit without a description or
>> a
>> > jira:
>> > >>>>>> https://github.com/apache/flink-statefun/pull/314
>> > >>>>>>
>> > >>>>>>
>> > >>>>>> On Wed, 26 Oct 2022 at 19:54, Galen Warren <
>> galen@cvillewarrens.com
>> > >
>> > >>>>>> wrote:
>> > >>>>>>
>> > >>>>>>> Hi Gordon (and others),
>> > >>>>>>>
>> > >>>>>>> I'm also using this project for stateful messaging, including
>> > >>>>>>> messaging
>> > >>>>>>> among functions.
>> > >>>>>>>
>> > >>>>>>> I've contributed a small amount of code in the past and have
>> also
>> > >>>>>>> enabled
>> > >>>>>>> Flink 1.15 compatibility in a local fork, so I might be able to
>> > help
>> > >>>>>>> out
>> > >>>>>>> here.
>> > >>>>>>>
>> > >>>>>>> Thanks,
>> > >>>>>>> Galen
>> > >>>>>>>
>> > >>>>>>> On Wed, Oct 26, 2022 at 1:34 PM Ken Krugler <
>> > >>>>>>> kkrugler_lists@transpac.com>
>> > >>>>>>> wrote:
>> > >>>>>>>
>> > >>>>>>> > Hi Gordon,
>> > >>>>>>> >
>> > >>>>>>> > We’re using it for stateful messaging, and also calling remote
>> > >>>>>>> > Python-based functions.
>> > >>>>>>> >
>> > >>>>>>> > So yes, also very interested in what is going to happen with
>> the
>> > >>>>>>> this
>> > >>>>>>> > subproject in the future.
>> > >>>>>>> >
>> > >>>>>>> > — Ken
>> > >>>>>>> >
>> > >>>>>>> >
>> > >>>>>>> >
>> > >>>>>>> > > Begin forwarded message:
>> > >>>>>>> > >
>> > >>>>>>> > > From: "Tzu-Li (Gordon) Tai" <tz...@apache.org>
>> > >>>>>>> > > Subject: Re: Stateful Functions with Flink 1.15 and onwards
>> > >>>>>>> > > Date: October 26, 2022 at 10:25:26 AM PDT
>> > >>>>>>> > > To: dev@flink.apache.org
>> > >>>>>>> > > Reply-To: dev@flink.apache.org
>> > >>>>>>> > >
>> > >>>>>>> > > Hi Filip,
>> > >>>>>>> > >
>> > >>>>>>> > > Thanks for bringing this up.
>> > >>>>>>> > >
>> > >>>>>>> > > The hard truth is that committers who were previously
>> active on
>> > >>>>>>> the
>> > >>>>>>> > > StateFun subproject, including myself, all currently have
>> other
>> > >>>>>>> focuses.
>> > >>>>>>> > > Indeed, we may need to discuss with the community on how to
>> > >>>>>>> proceed if
>> > >>>>>>> > > there seems to be no continued committer coverage.
>> > >>>>>>> > >
>> > >>>>>>> > > If it's just a matter of upgrading the supported Flink
>> version,
>> > >>>>>>> I'm still
>> > >>>>>>> > > familiar enough with the subproject to probably be able to
>> > drive
>> > >>>>>>> this (or
>> > >>>>>>> > > if your team is up to it, I can assist you on that).
>> > >>>>>>> > >
>> > >>>>>>> > > For the long-term, as a data point I'm curious to see how
>> many
>> > >>>>>>> users are
>> > >>>>>>> > > using StateFun in production today, and how you're using it?
>> > >>>>>>> > >
>> > >>>>>>> > > - Do your applications have arbitrary / cyclic /
>> > bi-directional
>> > >>>>>>> > > messaging between individual functions?
>> > >>>>>>> > > - Or are you utilizing StateFun simply to allow your
>> stateful
>> > >>>>>>> functions
>> > >>>>>>> > > to run remotely as separate processes?
>> > >>>>>>> > >
>> > >>>>>>> > > If the majority is only the latter category, there might be
>> a
>> > >>>>>>> case to
>> > >>>>>>> > > support remote functions natively in Flink (which has been a
>> > >>>>>>> discussion
>> > >>>>>>> > in
>> > >>>>>>> > > the past).
>> > >>>>>>> > >
>> > >>>>>>> > > Thanks,
>> > >>>>>>> > > Gordon
>> > >>>>>>> > >
>> > >>>>>>> > > On Wed, Oct 26, 2022 at 3:30 AM Filip Karnicki <
>> > >>>>>>> filip.karnicki@gmail.com
>> > >>>>>>> > >
>> > >>>>>>> > > wrote:
>> > >>>>>>> > >
>> > >>>>>>> > >> Hi, I noticed that the development on stateful functions
>> has
>> > >>>>>>> come to a
>> > >>>>>>> > bit
>> > >>>>>>> > >> of a halt, with a pull request to update statefun to use
>> Flink
>> > >>>>>>> 1.15
>> > >>>>>>> > being
>> > >>>>>>> > >> in the `open` state since May 2022.
>> > >>>>>>> > >>
>> > >>>>>>> > >> What do we think is the future of this sub-project?
>> > >>>>>>> > >>
>> > >>>>>>> > >> The background to this question is that my team is on a
>> shared
>> > >>>>>>> Flink
>> > >>>>>>> > >> cluster which will soon be upgrading to Flink 1.15. If I
>> need
>> > to
>> > >>>>>>> > re-write
>> > >>>>>>> > >> all our code as a native Flink job (rather than a remote
>> > >>>>>>> stateful
>> > >>>>>>> > function)
>> > >>>>>>> > >> then I need to get started right away.
>> > >>>>>>> > >>
>> > >>>>>>> > >> Many thanks,
>> > >>>>>>> > >> Fil
>> > >>>>>>> > >>
>> > >>>>>>> >
>> > >>>>>>> > --------------------------
>> > >>>>>>> > Ken Krugler
>> > >>>>>>> > http://www.scaleunlimited.com
>> > >>>>>>> > Custom big data solutions
>> > >>>>>>> > Flink, Pinot, Solr, Elasticsearch
>> > >>>>>>> >
>> > >>>>>>> >
>> > >>>>>>> >
>> > >>>>>>> >
>> > >>>>>>>
>> > >>>>>>
>> >
>>
>
Re: Stateful Functions with Flink 1.15 and onwards
Posted by Galen Warren <ga...@cvillewarrens.com>.
Thanks Gordon and Filip, I appreciate your help on this one.
On Tue, Nov 1, 2022 at 1:07 PM Tzu-Li (Gordon) Tai <tz...@apache.org>
wrote:
> PR for upgrading to Flink 1.15.2 has been merged. Thanks for the efforts,
> Galen and Filip!
>
> We should be ready to kick off a new release for StateFun with the Flink
> version upgrade.
> I'll cut off a release branch now on apache/flink-statefun for
> release-3.3.x to move things forward.
> @Galen, @Filip if you want to, after the release branch is cut, you could
> probably upgrade the master branch to Flink 1.16.x as well.
>
> Afterwards we should decide who is available to drive the actual release
> process for 3.3.0.
> There's quite a few steps that would require committer write access.
> Unless someone else is up for this earlier, I'll have some availability
> towards the end of next week to help drive this.
>
> Thanks,
> Gordon
>
> On Mon, Oct 31, 2022 at 12:17 PM Galen Warren <ga...@cvillewarrens.com>
> wrote:
>
> > Yes, that makes sense.
> >
> > PR is here: [FLINK-29814][statefun] Change supported Flink version to
> > 1.15.2 by galenwarren · Pull Request #319 · apache/flink-statefun
> > (github.com) <https://github.com/apache/flink-statefun/pull/319>.
> >
> > On Mon, Oct 31, 2022 at 11:35 AM Till Rohrmann <tr...@apache.org>
> > wrote:
> >
> > > I think there might still be value in supporting 1.15 since not
> everyone
> > > upgrades Flink very fast. Hopefully, for Statefun the diff between
> Flink
> > > 1.15 and 1.16 boils down to changing the Flink dependencies.
> > >
> > > Cheers,
> > > Till
> > >
> > > On Mon, Oct 31, 2022 at 2:06 PM Galen Warren <ga...@cvillewarrens.com>
> > > wrote:
> > >
> > >> Sure thing. One question -- Flink 1.16 was just released a few days
> ago.
> > >> Should I support 1.15, or just go straight to 1.16?
> > >>
> > >> On Mon, Oct 31, 2022 at 8:49 AM Till Rohrmann <tr...@apache.org>
> > >> wrote:
> > >>
> > >>> Hi folks,
> > >>>
> > >>> if you can open a PR for supporting Flink 1.15 Galen, then this would
> > be
> > >>> awesome. I've assigned you to this ticket. The next thing after
> merging
> > >>> this PR would be creating a new StateFun release. Once we have merged
> > the
> > >>> PR, let's check who can help with it the fastest.
> > >>>
> > >>> Cheers,
> > >>> Till
> > >>>
> > >>> On Mon, Oct 31, 2022 at 1:10 PM Galen Warren <
> galen@cvillewarrens.com>
> > >>> wrote:
> > >>>
> > >>>> Yes, I could do that.
> > >>>>
> > >>>> On Mon, Oct 31, 2022 at 7:48 AM Filip Karnicki <
> > >>>> filip.karnicki@gmail.com> wrote:
> > >>>>
> > >>>>> Hi All
> > >>>>>
> > >>>>> So what's the play here?
> > >>>>>
> > >>>>> Galen, what do you think about taking this on? Perhaps ++Till would
> > >>>>> assign this jira to you (with your permission) given he's helped me
> > out
> > >>>>> with statefun work before
> > >>>>> https://issues.apache.org/jira/browse/FLINK-29814
> > >>>>>
> > >>>>> I can try to move to move statefun to flink 1.16 when it's out
> > >>>>>
> > >>>>>
> > >>>>> Kind regards
> > >>>>> Fil
> > >>>>>
> > >>>>> On Thu, 27 Oct 2022 at 10:02, Filip Karnicki <
> > filip.karnicki@gmail.com>
> > >>>>> wrote:
> > >>>>>
> > >>>>>> Hi All
> > >>>>>>
> > >>>>>> Our use case is that we need to process elements for the same key
> > >>>>>> sequentially, and this processing involves async operations.
> > >>>>>>
> > >>>>>> If any part of the processing fails, we store the offending and
> all
> > >>>>>> subsequent incoming messages for that key in the state and not
> > process any
> > >>>>>> further messages for that key, until a retry succeeds or a human
> > sends a
> > >>>>>> 'skip' command message.
> > >>>>>>
> > >>>>>> diagram:
> > >>>>>>
> >
> https://mermaid.live/edit#pako:eNplkL1uwzAMhF-F0JQADrp76FR06tSOcQfWom3V-nFFqoUR591L20mXaqAOxHd3AC-mTZZMbfqM0wAvr00EfS62KbjYn-8C6Jui8DucTo_wmT4Oz97FEVhQqCtxXR13q_IBo-XzXWyehUc3LSu2Uyq2qIXpq2iyQ-9nmCjDSPMCmUISOuwfaEErLsVbw2272VOEDp0vmSqw5HEmC4GYsSeQpKjkv7j_buQ5tjAV4YeehOHHyQDsLAF1HbXCCyQZKB-2CTyzUOCjqUygHNBZPdxljW2MAoEaU6u0mMfGNPGqHBZJb1piasmFKlMmqxd7cqj3Dqbu0DNdfwHTGoek
> > >>>>>> mermaid (in case mermaid.live goes down in the future):
> > >>>>>> graph LR
> > >>>>>> incoming[incoming events] --> job(Flink statefun job)
> > >>>>>> commands[commands] -->|skip| job
> > >>>>>> job --> |sequentially per key| remote(remote function)
> > >>>>>> remote --> |on failure, delayed message to retry| remote
> > >>>>>> remote --> |async puts/gets with side effects| other(other
> > >>>>>> systems)
> > >>>>>>
> > >>>>>> Having the processing happen outside of Flink is nice-to-have from
> > an
> > >>>>>> independent scalability point of view, but is not strictly
> required.
> > >>>>>>
> > >>>>>> So long story short - no cyclic messaging, but also no way I can
> > >>>>>> think of to use existing native Flink operators like async i/o
> > (which when
> > >>>>>> I last checked a few years back didn't have access to keyed state)
> > >>>>>>
> > >>>>>>
> > >>>>>> P.S. Please note that there is already a pull request that has
> > >>>>>> something to do wtih Flink 1.15, albeit without a description or a
> > jira:
> > >>>>>> https://github.com/apache/flink-statefun/pull/314
> > >>>>>>
> > >>>>>>
> > >>>>>> On Wed, 26 Oct 2022 at 19:54, Galen Warren <
> galen@cvillewarrens.com
> > >
> > >>>>>> wrote:
> > >>>>>>
> > >>>>>>> Hi Gordon (and others),
> > >>>>>>>
> > >>>>>>> I'm also using this project for stateful messaging, including
> > >>>>>>> messaging
> > >>>>>>> among functions.
> > >>>>>>>
> > >>>>>>> I've contributed a small amount of code in the past and have also
> > >>>>>>> enabled
> > >>>>>>> Flink 1.15 compatibility in a local fork, so I might be able to
> > help
> > >>>>>>> out
> > >>>>>>> here.
> > >>>>>>>
> > >>>>>>> Thanks,
> > >>>>>>> Galen
> > >>>>>>>
> > >>>>>>> On Wed, Oct 26, 2022 at 1:34 PM Ken Krugler <
> > >>>>>>> kkrugler_lists@transpac.com>
> > >>>>>>> wrote:
> > >>>>>>>
> > >>>>>>> > Hi Gordon,
> > >>>>>>> >
> > >>>>>>> > We’re using it for stateful messaging, and also calling remote
> > >>>>>>> > Python-based functions.
> > >>>>>>> >
> > >>>>>>> > So yes, also very interested in what is going to happen with
> the
> > >>>>>>> this
> > >>>>>>> > subproject in the future.
> > >>>>>>> >
> > >>>>>>> > — Ken
> > >>>>>>> >
> > >>>>>>> >
> > >>>>>>> >
> > >>>>>>> > > Begin forwarded message:
> > >>>>>>> > >
> > >>>>>>> > > From: "Tzu-Li (Gordon) Tai" <tz...@apache.org>
> > >>>>>>> > > Subject: Re: Stateful Functions with Flink 1.15 and onwards
> > >>>>>>> > > Date: October 26, 2022 at 10:25:26 AM PDT
> > >>>>>>> > > To: dev@flink.apache.org
> > >>>>>>> > > Reply-To: dev@flink.apache.org
> > >>>>>>> > >
> > >>>>>>> > > Hi Filip,
> > >>>>>>> > >
> > >>>>>>> > > Thanks for bringing this up.
> > >>>>>>> > >
> > >>>>>>> > > The hard truth is that committers who were previously active
> on
> > >>>>>>> the
> > >>>>>>> > > StateFun subproject, including myself, all currently have
> other
> > >>>>>>> focuses.
> > >>>>>>> > > Indeed, we may need to discuss with the community on how to
> > >>>>>>> proceed if
> > >>>>>>> > > there seems to be no continued committer coverage.
> > >>>>>>> > >
> > >>>>>>> > > If it's just a matter of upgrading the supported Flink
> version,
> > >>>>>>> I'm still
> > >>>>>>> > > familiar enough with the subproject to probably be able to
> > drive
> > >>>>>>> this (or
> > >>>>>>> > > if your team is up to it, I can assist you on that).
> > >>>>>>> > >
> > >>>>>>> > > For the long-term, as a data point I'm curious to see how
> many
> > >>>>>>> users are
> > >>>>>>> > > using StateFun in production today, and how you're using it?
> > >>>>>>> > >
> > >>>>>>> > > - Do your applications have arbitrary / cyclic /
> > bi-directional
> > >>>>>>> > > messaging between individual functions?
> > >>>>>>> > > - Or are you utilizing StateFun simply to allow your
> stateful
> > >>>>>>> functions
> > >>>>>>> > > to run remotely as separate processes?
> > >>>>>>> > >
> > >>>>>>> > > If the majority is only the latter category, there might be a
> > >>>>>>> case to
> > >>>>>>> > > support remote functions natively in Flink (which has been a
> > >>>>>>> discussion
> > >>>>>>> > in
> > >>>>>>> > > the past).
> > >>>>>>> > >
> > >>>>>>> > > Thanks,
> > >>>>>>> > > Gordon
> > >>>>>>> > >
> > >>>>>>> > > On Wed, Oct 26, 2022 at 3:30 AM Filip Karnicki <
> > >>>>>>> filip.karnicki@gmail.com
> > >>>>>>> > >
> > >>>>>>> > > wrote:
> > >>>>>>> > >
> > >>>>>>> > >> Hi, I noticed that the development on stateful functions has
> > >>>>>>> come to a
> > >>>>>>> > bit
> > >>>>>>> > >> of a halt, with a pull request to update statefun to use
> Flink
> > >>>>>>> 1.15
> > >>>>>>> > being
> > >>>>>>> > >> in the `open` state since May 2022.
> > >>>>>>> > >>
> > >>>>>>> > >> What do we think is the future of this sub-project?
> > >>>>>>> > >>
> > >>>>>>> > >> The background to this question is that my team is on a
> shared
> > >>>>>>> Flink
> > >>>>>>> > >> cluster which will soon be upgrading to Flink 1.15. If I
> need
> > to
> > >>>>>>> > re-write
> > >>>>>>> > >> all our code as a native Flink job (rather than a remote
> > >>>>>>> stateful
> > >>>>>>> > function)
> > >>>>>>> > >> then I need to get started right away.
> > >>>>>>> > >>
> > >>>>>>> > >> Many thanks,
> > >>>>>>> > >> Fil
> > >>>>>>> > >>
> > >>>>>>> >
> > >>>>>>> > --------------------------
> > >>>>>>> > Ken Krugler
> > >>>>>>> > http://www.scaleunlimited.com
> > >>>>>>> > Custom big data solutions
> > >>>>>>> > Flink, Pinot, Solr, Elasticsearch
> > >>>>>>> >
> > >>>>>>> >
> > >>>>>>> >
> > >>>>>>> >
> > >>>>>>>
> > >>>>>>
> >
>
Re: Stateful Functions with Flink 1.15 and onwards
Posted by "Tzu-Li (Gordon) Tai" <tz...@apache.org>.
PR for upgrading to Flink 1.15.2 has been merged. Thanks for the efforts,
Galen and Filip!
We should be ready to kick off a new release for StateFun with the Flink
version upgrade.
I'll cut off a release branch now on apache/flink-statefun for
release-3.3.x to move things forward.
@Galen, @Filip if you want to, after the release branch is cut, you could
probably upgrade the master branch to Flink 1.16.x as well.
Afterwards we should decide who is available to drive the actual release
process for 3.3.0.
There's quite a few steps that would require committer write access.
Unless someone else is up for this earlier, I'll have some availability
towards the end of next week to help drive this.
Thanks,
Gordon
On Mon, Oct 31, 2022 at 12:17 PM Galen Warren <ga...@cvillewarrens.com>
wrote:
> Yes, that makes sense.
>
> PR is here: [FLINK-29814][statefun] Change supported Flink version to
> 1.15.2 by galenwarren · Pull Request #319 · apache/flink-statefun
> (github.com) <https://github.com/apache/flink-statefun/pull/319>.
>
> On Mon, Oct 31, 2022 at 11:35 AM Till Rohrmann <tr...@apache.org>
> wrote:
>
> > I think there might still be value in supporting 1.15 since not everyone
> > upgrades Flink very fast. Hopefully, for Statefun the diff between Flink
> > 1.15 and 1.16 boils down to changing the Flink dependencies.
> >
> > Cheers,
> > Till
> >
> > On Mon, Oct 31, 2022 at 2:06 PM Galen Warren <ga...@cvillewarrens.com>
> > wrote:
> >
> >> Sure thing. One question -- Flink 1.16 was just released a few days ago.
> >> Should I support 1.15, or just go straight to 1.16?
> >>
> >> On Mon, Oct 31, 2022 at 8:49 AM Till Rohrmann <tr...@apache.org>
> >> wrote:
> >>
> >>> Hi folks,
> >>>
> >>> if you can open a PR for supporting Flink 1.15 Galen, then this would
> be
> >>> awesome. I've assigned you to this ticket. The next thing after merging
> >>> this PR would be creating a new StateFun release. Once we have merged
> the
> >>> PR, let's check who can help with it the fastest.
> >>>
> >>> Cheers,
> >>> Till
> >>>
> >>> On Mon, Oct 31, 2022 at 1:10 PM Galen Warren <ga...@cvillewarrens.com>
> >>> wrote:
> >>>
> >>>> Yes, I could do that.
> >>>>
> >>>> On Mon, Oct 31, 2022 at 7:48 AM Filip Karnicki <
> >>>> filip.karnicki@gmail.com> wrote:
> >>>>
> >>>>> Hi All
> >>>>>
> >>>>> So what's the play here?
> >>>>>
> >>>>> Galen, what do you think about taking this on? Perhaps ++Till would
> >>>>> assign this jira to you (with your permission) given he's helped me
> out
> >>>>> with statefun work before
> >>>>> https://issues.apache.org/jira/browse/FLINK-29814
> >>>>>
> >>>>> I can try to move to move statefun to flink 1.16 when it's out
> >>>>>
> >>>>>
> >>>>> Kind regards
> >>>>> Fil
> >>>>>
> >>>>> On Thu, 27 Oct 2022 at 10:02, Filip Karnicki <
> filip.karnicki@gmail.com>
> >>>>> wrote:
> >>>>>
> >>>>>> Hi All
> >>>>>>
> >>>>>> Our use case is that we need to process elements for the same key
> >>>>>> sequentially, and this processing involves async operations.
> >>>>>>
> >>>>>> If any part of the processing fails, we store the offending and all
> >>>>>> subsequent incoming messages for that key in the state and not
> process any
> >>>>>> further messages for that key, until a retry succeeds or a human
> sends a
> >>>>>> 'skip' command message.
> >>>>>>
> >>>>>> diagram:
> >>>>>>
> https://mermaid.live/edit#pako:eNplkL1uwzAMhF-F0JQADrp76FR06tSOcQfWom3V-nFFqoUR591L20mXaqAOxHd3AC-mTZZMbfqM0wAvr00EfS62KbjYn-8C6Jui8DucTo_wmT4Oz97FEVhQqCtxXR13q_IBo-XzXWyehUc3LSu2Uyq2qIXpq2iyQ-9nmCjDSPMCmUISOuwfaEErLsVbw2272VOEDp0vmSqw5HEmC4GYsSeQpKjkv7j_buQ5tjAV4YeehOHHyQDsLAF1HbXCCyQZKB-2CTyzUOCjqUygHNBZPdxljW2MAoEaU6u0mMfGNPGqHBZJb1piasmFKlMmqxd7cqj3Dqbu0DNdfwHTGoek
> >>>>>> mermaid (in case mermaid.live goes down in the future):
> >>>>>> graph LR
> >>>>>> incoming[incoming events] --> job(Flink statefun job)
> >>>>>> commands[commands] -->|skip| job
> >>>>>> job --> |sequentially per key| remote(remote function)
> >>>>>> remote --> |on failure, delayed message to retry| remote
> >>>>>> remote --> |async puts/gets with side effects| other(other
> >>>>>> systems)
> >>>>>>
> >>>>>> Having the processing happen outside of Flink is nice-to-have from
> an
> >>>>>> independent scalability point of view, but is not strictly required.
> >>>>>>
> >>>>>> So long story short - no cyclic messaging, but also no way I can
> >>>>>> think of to use existing native Flink operators like async i/o
> (which when
> >>>>>> I last checked a few years back didn't have access to keyed state)
> >>>>>>
> >>>>>>
> >>>>>> P.S. Please note that there is already a pull request that has
> >>>>>> something to do wtih Flink 1.15, albeit without a description or a
> jira:
> >>>>>> https://github.com/apache/flink-statefun/pull/314
> >>>>>>
> >>>>>>
> >>>>>> On Wed, 26 Oct 2022 at 19:54, Galen Warren <galen@cvillewarrens.com
> >
> >>>>>> wrote:
> >>>>>>
> >>>>>>> Hi Gordon (and others),
> >>>>>>>
> >>>>>>> I'm also using this project for stateful messaging, including
> >>>>>>> messaging
> >>>>>>> among functions.
> >>>>>>>
> >>>>>>> I've contributed a small amount of code in the past and have also
> >>>>>>> enabled
> >>>>>>> Flink 1.15 compatibility in a local fork, so I might be able to
> help
> >>>>>>> out
> >>>>>>> here.
> >>>>>>>
> >>>>>>> Thanks,
> >>>>>>> Galen
> >>>>>>>
> >>>>>>> On Wed, Oct 26, 2022 at 1:34 PM Ken Krugler <
> >>>>>>> kkrugler_lists@transpac.com>
> >>>>>>> wrote:
> >>>>>>>
> >>>>>>> > Hi Gordon,
> >>>>>>> >
> >>>>>>> > We’re using it for stateful messaging, and also calling remote
> >>>>>>> > Python-based functions.
> >>>>>>> >
> >>>>>>> > So yes, also very interested in what is going to happen with the
> >>>>>>> this
> >>>>>>> > subproject in the future.
> >>>>>>> >
> >>>>>>> > — Ken
> >>>>>>> >
> >>>>>>> >
> >>>>>>> >
> >>>>>>> > > Begin forwarded message:
> >>>>>>> > >
> >>>>>>> > > From: "Tzu-Li (Gordon) Tai" <tz...@apache.org>
> >>>>>>> > > Subject: Re: Stateful Functions with Flink 1.15 and onwards
> >>>>>>> > > Date: October 26, 2022 at 10:25:26 AM PDT
> >>>>>>> > > To: dev@flink.apache.org
> >>>>>>> > > Reply-To: dev@flink.apache.org
> >>>>>>> > >
> >>>>>>> > > Hi Filip,
> >>>>>>> > >
> >>>>>>> > > Thanks for bringing this up.
> >>>>>>> > >
> >>>>>>> > > The hard truth is that committers who were previously active on
> >>>>>>> the
> >>>>>>> > > StateFun subproject, including myself, all currently have other
> >>>>>>> focuses.
> >>>>>>> > > Indeed, we may need to discuss with the community on how to
> >>>>>>> proceed if
> >>>>>>> > > there seems to be no continued committer coverage.
> >>>>>>> > >
> >>>>>>> > > If it's just a matter of upgrading the supported Flink version,
> >>>>>>> I'm still
> >>>>>>> > > familiar enough with the subproject to probably be able to
> drive
> >>>>>>> this (or
> >>>>>>> > > if your team is up to it, I can assist you on that).
> >>>>>>> > >
> >>>>>>> > > For the long-term, as a data point I'm curious to see how many
> >>>>>>> users are
> >>>>>>> > > using StateFun in production today, and how you're using it?
> >>>>>>> > >
> >>>>>>> > > - Do your applications have arbitrary / cyclic /
> bi-directional
> >>>>>>> > > messaging between individual functions?
> >>>>>>> > > - Or are you utilizing StateFun simply to allow your stateful
> >>>>>>> functions
> >>>>>>> > > to run remotely as separate processes?
> >>>>>>> > >
> >>>>>>> > > If the majority is only the latter category, there might be a
> >>>>>>> case to
> >>>>>>> > > support remote functions natively in Flink (which has been a
> >>>>>>> discussion
> >>>>>>> > in
> >>>>>>> > > the past).
> >>>>>>> > >
> >>>>>>> > > Thanks,
> >>>>>>> > > Gordon
> >>>>>>> > >
> >>>>>>> > > On Wed, Oct 26, 2022 at 3:30 AM Filip Karnicki <
> >>>>>>> filip.karnicki@gmail.com
> >>>>>>> > >
> >>>>>>> > > wrote:
> >>>>>>> > >
> >>>>>>> > >> Hi, I noticed that the development on stateful functions has
> >>>>>>> come to a
> >>>>>>> > bit
> >>>>>>> > >> of a halt, with a pull request to update statefun to use Flink
> >>>>>>> 1.15
> >>>>>>> > being
> >>>>>>> > >> in the `open` state since May 2022.
> >>>>>>> > >>
> >>>>>>> > >> What do we think is the future of this sub-project?
> >>>>>>> > >>
> >>>>>>> > >> The background to this question is that my team is on a shared
> >>>>>>> Flink
> >>>>>>> > >> cluster which will soon be upgrading to Flink 1.15. If I need
> to
> >>>>>>> > re-write
> >>>>>>> > >> all our code as a native Flink job (rather than a remote
> >>>>>>> stateful
> >>>>>>> > function)
> >>>>>>> > >> then I need to get started right away.
> >>>>>>> > >>
> >>>>>>> > >> Many thanks,
> >>>>>>> > >> Fil
> >>>>>>> > >>
> >>>>>>> >
> >>>>>>> > --------------------------
> >>>>>>> > Ken Krugler
> >>>>>>> > http://www.scaleunlimited.com
> >>>>>>> > Custom big data solutions
> >>>>>>> > Flink, Pinot, Solr, Elasticsearch
> >>>>>>> >
> >>>>>>> >
> >>>>>>> >
> >>>>>>> >
> >>>>>>>
> >>>>>>
>
Re: Stateful Functions with Flink 1.15 and onwards
Posted by Galen Warren <ga...@cvillewarrens.com>.
Yes, that makes sense.
PR is here: [FLINK-29814][statefun] Change supported Flink version to
1.15.2 by galenwarren · Pull Request #319 · apache/flink-statefun
(github.com) <https://github.com/apache/flink-statefun/pull/319>.
On Mon, Oct 31, 2022 at 11:35 AM Till Rohrmann <tr...@apache.org> wrote:
> I think there might still be value in supporting 1.15 since not everyone
> upgrades Flink very fast. Hopefully, for Statefun the diff between Flink
> 1.15 and 1.16 boils down to changing the Flink dependencies.
>
> Cheers,
> Till
>
> On Mon, Oct 31, 2022 at 2:06 PM Galen Warren <ga...@cvillewarrens.com>
> wrote:
>
>> Sure thing. One question -- Flink 1.16 was just released a few days ago.
>> Should I support 1.15, or just go straight to 1.16?
>>
>> On Mon, Oct 31, 2022 at 8:49 AM Till Rohrmann <tr...@apache.org>
>> wrote:
>>
>>> Hi folks,
>>>
>>> if you can open a PR for supporting Flink 1.15 Galen, then this would be
>>> awesome. I've assigned you to this ticket. The next thing after merging
>>> this PR would be creating a new StateFun release. Once we have merged the
>>> PR, let's check who can help with it the fastest.
>>>
>>> Cheers,
>>> Till
>>>
>>> On Mon, Oct 31, 2022 at 1:10 PM Galen Warren <ga...@cvillewarrens.com>
>>> wrote:
>>>
>>>> Yes, I could do that.
>>>>
>>>> On Mon, Oct 31, 2022 at 7:48 AM Filip Karnicki <
>>>> filip.karnicki@gmail.com> wrote:
>>>>
>>>>> Hi All
>>>>>
>>>>> So what's the play here?
>>>>>
>>>>> Galen, what do you think about taking this on? Perhaps ++Till would
>>>>> assign this jira to you (with your permission) given he's helped me out
>>>>> with statefun work before
>>>>> https://issues.apache.org/jira/browse/FLINK-29814
>>>>>
>>>>> I can try to move to move statefun to flink 1.16 when it's out
>>>>>
>>>>>
>>>>> Kind regards
>>>>> Fil
>>>>>
>>>>> On Thu, 27 Oct 2022 at 10:02, Filip Karnicki <fi...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Hi All
>>>>>>
>>>>>> Our use case is that we need to process elements for the same key
>>>>>> sequentially, and this processing involves async operations.
>>>>>>
>>>>>> If any part of the processing fails, we store the offending and all
>>>>>> subsequent incoming messages for that key in the state and not process any
>>>>>> further messages for that key, until a retry succeeds or a human sends a
>>>>>> 'skip' command message.
>>>>>>
>>>>>> diagram:
>>>>>> https://mermaid.live/edit#pako:eNplkL1uwzAMhF-F0JQADrp76FR06tSOcQfWom3V-nFFqoUR591L20mXaqAOxHd3AC-mTZZMbfqM0wAvr00EfS62KbjYn-8C6Jui8DucTo_wmT4Oz97FEVhQqCtxXR13q_IBo-XzXWyehUc3LSu2Uyq2qIXpq2iyQ-9nmCjDSPMCmUISOuwfaEErLsVbw2272VOEDp0vmSqw5HEmC4GYsSeQpKjkv7j_buQ5tjAV4YeehOHHyQDsLAF1HbXCCyQZKB-2CTyzUOCjqUygHNBZPdxljW2MAoEaU6u0mMfGNPGqHBZJb1piasmFKlMmqxd7cqj3Dqbu0DNdfwHTGoek
>>>>>> mermaid (in case mermaid.live goes down in the future):
>>>>>> graph LR
>>>>>> incoming[incoming events] --> job(Flink statefun job)
>>>>>> commands[commands] -->|skip| job
>>>>>> job --> |sequentially per key| remote(remote function)
>>>>>> remote --> |on failure, delayed message to retry| remote
>>>>>> remote --> |async puts/gets with side effects| other(other
>>>>>> systems)
>>>>>>
>>>>>> Having the processing happen outside of Flink is nice-to-have from an
>>>>>> independent scalability point of view, but is not strictly required.
>>>>>>
>>>>>> So long story short - no cyclic messaging, but also no way I can
>>>>>> think of to use existing native Flink operators like async i/o (which when
>>>>>> I last checked a few years back didn't have access to keyed state)
>>>>>>
>>>>>>
>>>>>> P.S. Please note that there is already a pull request that has
>>>>>> something to do wtih Flink 1.15, albeit without a description or a jira:
>>>>>> https://github.com/apache/flink-statefun/pull/314
>>>>>>
>>>>>>
>>>>>> On Wed, 26 Oct 2022 at 19:54, Galen Warren <ga...@cvillewarrens.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi Gordon (and others),
>>>>>>>
>>>>>>> I'm also using this project for stateful messaging, including
>>>>>>> messaging
>>>>>>> among functions.
>>>>>>>
>>>>>>> I've contributed a small amount of code in the past and have also
>>>>>>> enabled
>>>>>>> Flink 1.15 compatibility in a local fork, so I might be able to help
>>>>>>> out
>>>>>>> here.
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Galen
>>>>>>>
>>>>>>> On Wed, Oct 26, 2022 at 1:34 PM Ken Krugler <
>>>>>>> kkrugler_lists@transpac.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>> > Hi Gordon,
>>>>>>> >
>>>>>>> > We’re using it for stateful messaging, and also calling remote
>>>>>>> > Python-based functions.
>>>>>>> >
>>>>>>> > So yes, also very interested in what is going to happen with the
>>>>>>> this
>>>>>>> > subproject in the future.
>>>>>>> >
>>>>>>> > — Ken
>>>>>>> >
>>>>>>> >
>>>>>>> >
>>>>>>> > > Begin forwarded message:
>>>>>>> > >
>>>>>>> > > From: "Tzu-Li (Gordon) Tai" <tz...@apache.org>
>>>>>>> > > Subject: Re: Stateful Functions with Flink 1.15 and onwards
>>>>>>> > > Date: October 26, 2022 at 10:25:26 AM PDT
>>>>>>> > > To: dev@flink.apache.org
>>>>>>> > > Reply-To: dev@flink.apache.org
>>>>>>> > >
>>>>>>> > > Hi Filip,
>>>>>>> > >
>>>>>>> > > Thanks for bringing this up.
>>>>>>> > >
>>>>>>> > > The hard truth is that committers who were previously active on
>>>>>>> the
>>>>>>> > > StateFun subproject, including myself, all currently have other
>>>>>>> focuses.
>>>>>>> > > Indeed, we may need to discuss with the community on how to
>>>>>>> proceed if
>>>>>>> > > there seems to be no continued committer coverage.
>>>>>>> > >
>>>>>>> > > If it's just a matter of upgrading the supported Flink version,
>>>>>>> I'm still
>>>>>>> > > familiar enough with the subproject to probably be able to drive
>>>>>>> this (or
>>>>>>> > > if your team is up to it, I can assist you on that).
>>>>>>> > >
>>>>>>> > > For the long-term, as a data point I'm curious to see how many
>>>>>>> users are
>>>>>>> > > using StateFun in production today, and how you're using it?
>>>>>>> > >
>>>>>>> > > - Do your applications have arbitrary / cyclic / bi-directional
>>>>>>> > > messaging between individual functions?
>>>>>>> > > - Or are you utilizing StateFun simply to allow your stateful
>>>>>>> functions
>>>>>>> > > to run remotely as separate processes?
>>>>>>> > >
>>>>>>> > > If the majority is only the latter category, there might be a
>>>>>>> case to
>>>>>>> > > support remote functions natively in Flink (which has been a
>>>>>>> discussion
>>>>>>> > in
>>>>>>> > > the past).
>>>>>>> > >
>>>>>>> > > Thanks,
>>>>>>> > > Gordon
>>>>>>> > >
>>>>>>> > > On Wed, Oct 26, 2022 at 3:30 AM Filip Karnicki <
>>>>>>> filip.karnicki@gmail.com
>>>>>>> > >
>>>>>>> > > wrote:
>>>>>>> > >
>>>>>>> > >> Hi, I noticed that the development on stateful functions has
>>>>>>> come to a
>>>>>>> > bit
>>>>>>> > >> of a halt, with a pull request to update statefun to use Flink
>>>>>>> 1.15
>>>>>>> > being
>>>>>>> > >> in the `open` state since May 2022.
>>>>>>> > >>
>>>>>>> > >> What do we think is the future of this sub-project?
>>>>>>> > >>
>>>>>>> > >> The background to this question is that my team is on a shared
>>>>>>> Flink
>>>>>>> > >> cluster which will soon be upgrading to Flink 1.15. If I need to
>>>>>>> > re-write
>>>>>>> > >> all our code as a native Flink job (rather than a remote
>>>>>>> stateful
>>>>>>> > function)
>>>>>>> > >> then I need to get started right away.
>>>>>>> > >>
>>>>>>> > >> Many thanks,
>>>>>>> > >> Fil
>>>>>>> > >>
>>>>>>> >
>>>>>>> > --------------------------
>>>>>>> > Ken Krugler
>>>>>>> > http://www.scaleunlimited.com
>>>>>>> > Custom big data solutions
>>>>>>> > Flink, Pinot, Solr, Elasticsearch
>>>>>>> >
>>>>>>> >
>>>>>>> >
>>>>>>> >
>>>>>>>
>>>>>>
Re: Stateful Functions with Flink 1.15 and onwards
Posted by Till Rohrmann <tr...@apache.org>.
I think there might still be value in supporting 1.15 since not everyone
upgrades Flink very fast. Hopefully, for Statefun the diff between Flink
1.15 and 1.16 boils down to changing the Flink dependencies.
Cheers,
Till
On Mon, Oct 31, 2022 at 2:06 PM Galen Warren <ga...@cvillewarrens.com>
wrote:
> Sure thing. One question -- Flink 1.16 was just released a few days ago.
> Should I support 1.15, or just go straight to 1.16?
>
> On Mon, Oct 31, 2022 at 8:49 AM Till Rohrmann <tr...@apache.org>
> wrote:
>
>> Hi folks,
>>
>> if you can open a PR for supporting Flink 1.15 Galen, then this would be
>> awesome. I've assigned you to this ticket. The next thing after merging
>> this PR would be creating a new StateFun release. Once we have merged the
>> PR, let's check who can help with it the fastest.
>>
>> Cheers,
>> Till
>>
>> On Mon, Oct 31, 2022 at 1:10 PM Galen Warren <ga...@cvillewarrens.com>
>> wrote:
>>
>>> Yes, I could do that.
>>>
>>> On Mon, Oct 31, 2022 at 7:48 AM Filip Karnicki <fi...@gmail.com>
>>> wrote:
>>>
>>>> Hi All
>>>>
>>>> So what's the play here?
>>>>
>>>> Galen, what do you think about taking this on? Perhaps ++Till would
>>>> assign this jira to you (with your permission) given he's helped me out
>>>> with statefun work before
>>>> https://issues.apache.org/jira/browse/FLINK-29814
>>>>
>>>> I can try to move to move statefun to flink 1.16 when it's out
>>>>
>>>>
>>>> Kind regards
>>>> Fil
>>>>
>>>> On Thu, 27 Oct 2022 at 10:02, Filip Karnicki <fi...@gmail.com>
>>>> wrote:
>>>>
>>>>> Hi All
>>>>>
>>>>> Our use case is that we need to process elements for the same key
>>>>> sequentially, and this processing involves async operations.
>>>>>
>>>>> If any part of the processing fails, we store the offending and all
>>>>> subsequent incoming messages for that key in the state and not process any
>>>>> further messages for that key, until a retry succeeds or a human sends a
>>>>> 'skip' command message.
>>>>>
>>>>> diagram:
>>>>> https://mermaid.live/edit#pako:eNplkL1uwzAMhF-F0JQADrp76FR06tSOcQfWom3V-nFFqoUR591L20mXaqAOxHd3AC-mTZZMbfqM0wAvr00EfS62KbjYn-8C6Jui8DucTo_wmT4Oz97FEVhQqCtxXR13q_IBo-XzXWyehUc3LSu2Uyq2qIXpq2iyQ-9nmCjDSPMCmUISOuwfaEErLsVbw2272VOEDp0vmSqw5HEmC4GYsSeQpKjkv7j_buQ5tjAV4YeehOHHyQDsLAF1HbXCCyQZKB-2CTyzUOCjqUygHNBZPdxljW2MAoEaU6u0mMfGNPGqHBZJb1piasmFKlMmqxd7cqj3Dqbu0DNdfwHTGoek
>>>>> mermaid (in case mermaid.live goes down in the future):
>>>>> graph LR
>>>>> incoming[incoming events] --> job(Flink statefun job)
>>>>> commands[commands] -->|skip| job
>>>>> job --> |sequentially per key| remote(remote function)
>>>>> remote --> |on failure, delayed message to retry| remote
>>>>> remote --> |async puts/gets with side effects| other(other
>>>>> systems)
>>>>>
>>>>> Having the processing happen outside of Flink is nice-to-have from an
>>>>> independent scalability point of view, but is not strictly required.
>>>>>
>>>>> So long story short - no cyclic messaging, but also no way I can think
>>>>> of to use existing native Flink operators like async i/o (which when I last
>>>>> checked a few years back didn't have access to keyed state)
>>>>>
>>>>>
>>>>> P.S. Please note that there is already a pull request that has
>>>>> something to do wtih Flink 1.15, albeit without a description or a jira:
>>>>> https://github.com/apache/flink-statefun/pull/314
>>>>>
>>>>>
>>>>> On Wed, 26 Oct 2022 at 19:54, Galen Warren <ga...@cvillewarrens.com>
>>>>> wrote:
>>>>>
>>>>>> Hi Gordon (and others),
>>>>>>
>>>>>> I'm also using this project for stateful messaging, including
>>>>>> messaging
>>>>>> among functions.
>>>>>>
>>>>>> I've contributed a small amount of code in the past and have also
>>>>>> enabled
>>>>>> Flink 1.15 compatibility in a local fork, so I might be able to help
>>>>>> out
>>>>>> here.
>>>>>>
>>>>>> Thanks,
>>>>>> Galen
>>>>>>
>>>>>> On Wed, Oct 26, 2022 at 1:34 PM Ken Krugler <
>>>>>> kkrugler_lists@transpac.com>
>>>>>> wrote:
>>>>>>
>>>>>> > Hi Gordon,
>>>>>> >
>>>>>> > We’re using it for stateful messaging, and also calling remote
>>>>>> > Python-based functions.
>>>>>> >
>>>>>> > So yes, also very interested in what is going to happen with the
>>>>>> this
>>>>>> > subproject in the future.
>>>>>> >
>>>>>> > — Ken
>>>>>> >
>>>>>> >
>>>>>> >
>>>>>> > > Begin forwarded message:
>>>>>> > >
>>>>>> > > From: "Tzu-Li (Gordon) Tai" <tz...@apache.org>
>>>>>> > > Subject: Re: Stateful Functions with Flink 1.15 and onwards
>>>>>> > > Date: October 26, 2022 at 10:25:26 AM PDT
>>>>>> > > To: dev@flink.apache.org
>>>>>> > > Reply-To: dev@flink.apache.org
>>>>>> > >
>>>>>> > > Hi Filip,
>>>>>> > >
>>>>>> > > Thanks for bringing this up.
>>>>>> > >
>>>>>> > > The hard truth is that committers who were previously active on
>>>>>> the
>>>>>> > > StateFun subproject, including myself, all currently have other
>>>>>> focuses.
>>>>>> > > Indeed, we may need to discuss with the community on how to
>>>>>> proceed if
>>>>>> > > there seems to be no continued committer coverage.
>>>>>> > >
>>>>>> > > If it's just a matter of upgrading the supported Flink version,
>>>>>> I'm still
>>>>>> > > familiar enough with the subproject to probably be able to drive
>>>>>> this (or
>>>>>> > > if your team is up to it, I can assist you on that).
>>>>>> > >
>>>>>> > > For the long-term, as a data point I'm curious to see how many
>>>>>> users are
>>>>>> > > using StateFun in production today, and how you're using it?
>>>>>> > >
>>>>>> > > - Do your applications have arbitrary / cyclic / bi-directional
>>>>>> > > messaging between individual functions?
>>>>>> > > - Or are you utilizing StateFun simply to allow your stateful
>>>>>> functions
>>>>>> > > to run remotely as separate processes?
>>>>>> > >
>>>>>> > > If the majority is only the latter category, there might be a
>>>>>> case to
>>>>>> > > support remote functions natively in Flink (which has been a
>>>>>> discussion
>>>>>> > in
>>>>>> > > the past).
>>>>>> > >
>>>>>> > > Thanks,
>>>>>> > > Gordon
>>>>>> > >
>>>>>> > > On Wed, Oct 26, 2022 at 3:30 AM Filip Karnicki <
>>>>>> filip.karnicki@gmail.com
>>>>>> > >
>>>>>> > > wrote:
>>>>>> > >
>>>>>> > >> Hi, I noticed that the development on stateful functions has
>>>>>> come to a
>>>>>> > bit
>>>>>> > >> of a halt, with a pull request to update statefun to use Flink
>>>>>> 1.15
>>>>>> > being
>>>>>> > >> in the `open` state since May 2022.
>>>>>> > >>
>>>>>> > >> What do we think is the future of this sub-project?
>>>>>> > >>
>>>>>> > >> The background to this question is that my team is on a shared
>>>>>> Flink
>>>>>> > >> cluster which will soon be upgrading to Flink 1.15. If I need to
>>>>>> > re-write
>>>>>> > >> all our code as a native Flink job (rather than a remote stateful
>>>>>> > function)
>>>>>> > >> then I need to get started right away.
>>>>>> > >>
>>>>>> > >> Many thanks,
>>>>>> > >> Fil
>>>>>> > >>
>>>>>> >
>>>>>> > --------------------------
>>>>>> > Ken Krugler
>>>>>> > http://www.scaleunlimited.com
>>>>>> > Custom big data solutions
>>>>>> > Flink, Pinot, Solr, Elasticsearch
>>>>>> >
>>>>>> >
>>>>>> >
>>>>>> >
>>>>>>
>>>>>
Re: Stateful Functions with Flink 1.15 and onwards
Posted by Galen Warren <ga...@cvillewarrens.com>.
Sure thing. One question -- Flink 1.16 was just released a few days ago.
Should I support 1.15, or just go straight to 1.16?
On Mon, Oct 31, 2022 at 8:49 AM Till Rohrmann <tr...@apache.org> wrote:
> Hi folks,
>
> if you can open a PR for supporting Flink 1.15 Galen, then this would be
> awesome. I've assigned you to this ticket. The next thing after merging
> this PR would be creating a new StateFun release. Once we have merged the
> PR, let's check who can help with it the fastest.
>
> Cheers,
> Till
>
> On Mon, Oct 31, 2022 at 1:10 PM Galen Warren <ga...@cvillewarrens.com>
> wrote:
>
>> Yes, I could do that.
>>
>> On Mon, Oct 31, 2022 at 7:48 AM Filip Karnicki <fi...@gmail.com>
>> wrote:
>>
>>> Hi All
>>>
>>> So what's the play here?
>>>
>>> Galen, what do you think about taking this on? Perhaps ++Till would
>>> assign this jira to you (with your permission) given he's helped me out
>>> with statefun work before
>>> https://issues.apache.org/jira/browse/FLINK-29814
>>>
>>> I can try to move to move statefun to flink 1.16 when it's out
>>>
>>>
>>> Kind regards
>>> Fil
>>>
>>> On Thu, 27 Oct 2022 at 10:02, Filip Karnicki <fi...@gmail.com>
>>> wrote:
>>>
>>>> Hi All
>>>>
>>>> Our use case is that we need to process elements for the same key
>>>> sequentially, and this processing involves async operations.
>>>>
>>>> If any part of the processing fails, we store the offending and all
>>>> subsequent incoming messages for that key in the state and not process any
>>>> further messages for that key, until a retry succeeds or a human sends a
>>>> 'skip' command message.
>>>>
>>>> diagram:
>>>> https://mermaid.live/edit#pako:eNplkL1uwzAMhF-F0JQADrp76FR06tSOcQfWom3V-nFFqoUR591L20mXaqAOxHd3AC-mTZZMbfqM0wAvr00EfS62KbjYn-8C6Jui8DucTo_wmT4Oz97FEVhQqCtxXR13q_IBo-XzXWyehUc3LSu2Uyq2qIXpq2iyQ-9nmCjDSPMCmUISOuwfaEErLsVbw2272VOEDp0vmSqw5HEmC4GYsSeQpKjkv7j_buQ5tjAV4YeehOHHyQDsLAF1HbXCCyQZKB-2CTyzUOCjqUygHNBZPdxljW2MAoEaU6u0mMfGNPGqHBZJb1piasmFKlMmqxd7cqj3Dqbu0DNdfwHTGoek
>>>> mermaid (in case mermaid.live goes down in the future):
>>>> graph LR
>>>> incoming[incoming events] --> job(Flink statefun job)
>>>> commands[commands] -->|skip| job
>>>> job --> |sequentially per key| remote(remote function)
>>>> remote --> |on failure, delayed message to retry| remote
>>>> remote --> |async puts/gets with side effects| other(other systems)
>>>>
>>>> Having the processing happen outside of Flink is nice-to-have from an
>>>> independent scalability point of view, but is not strictly required.
>>>>
>>>> So long story short - no cyclic messaging, but also no way I can think
>>>> of to use existing native Flink operators like async i/o (which when I last
>>>> checked a few years back didn't have access to keyed state)
>>>>
>>>>
>>>> P.S. Please note that there is already a pull request that has
>>>> something to do wtih Flink 1.15, albeit without a description or a jira:
>>>> https://github.com/apache/flink-statefun/pull/314
>>>>
>>>>
>>>> On Wed, 26 Oct 2022 at 19:54, Galen Warren <ga...@cvillewarrens.com>
>>>> wrote:
>>>>
>>>>> Hi Gordon (and others),
>>>>>
>>>>> I'm also using this project for stateful messaging, including messaging
>>>>> among functions.
>>>>>
>>>>> I've contributed a small amount of code in the past and have also
>>>>> enabled
>>>>> Flink 1.15 compatibility in a local fork, so I might be able to help
>>>>> out
>>>>> here.
>>>>>
>>>>> Thanks,
>>>>> Galen
>>>>>
>>>>> On Wed, Oct 26, 2022 at 1:34 PM Ken Krugler <
>>>>> kkrugler_lists@transpac.com>
>>>>> wrote:
>>>>>
>>>>> > Hi Gordon,
>>>>> >
>>>>> > We’re using it for stateful messaging, and also calling remote
>>>>> > Python-based functions.
>>>>> >
>>>>> > So yes, also very interested in what is going to happen with the this
>>>>> > subproject in the future.
>>>>> >
>>>>> > — Ken
>>>>> >
>>>>> >
>>>>> >
>>>>> > > Begin forwarded message:
>>>>> > >
>>>>> > > From: "Tzu-Li (Gordon) Tai" <tz...@apache.org>
>>>>> > > Subject: Re: Stateful Functions with Flink 1.15 and onwards
>>>>> > > Date: October 26, 2022 at 10:25:26 AM PDT
>>>>> > > To: dev@flink.apache.org
>>>>> > > Reply-To: dev@flink.apache.org
>>>>> > >
>>>>> > > Hi Filip,
>>>>> > >
>>>>> > > Thanks for bringing this up.
>>>>> > >
>>>>> > > The hard truth is that committers who were previously active on the
>>>>> > > StateFun subproject, including myself, all currently have other
>>>>> focuses.
>>>>> > > Indeed, we may need to discuss with the community on how to
>>>>> proceed if
>>>>> > > there seems to be no continued committer coverage.
>>>>> > >
>>>>> > > If it's just a matter of upgrading the supported Flink version,
>>>>> I'm still
>>>>> > > familiar enough with the subproject to probably be able to drive
>>>>> this (or
>>>>> > > if your team is up to it, I can assist you on that).
>>>>> > >
>>>>> > > For the long-term, as a data point I'm curious to see how many
>>>>> users are
>>>>> > > using StateFun in production today, and how you're using it?
>>>>> > >
>>>>> > > - Do your applications have arbitrary / cyclic / bi-directional
>>>>> > > messaging between individual functions?
>>>>> > > - Or are you utilizing StateFun simply to allow your stateful
>>>>> functions
>>>>> > > to run remotely as separate processes?
>>>>> > >
>>>>> > > If the majority is only the latter category, there might be a case
>>>>> to
>>>>> > > support remote functions natively in Flink (which has been a
>>>>> discussion
>>>>> > in
>>>>> > > the past).
>>>>> > >
>>>>> > > Thanks,
>>>>> > > Gordon
>>>>> > >
>>>>> > > On Wed, Oct 26, 2022 at 3:30 AM Filip Karnicki <
>>>>> filip.karnicki@gmail.com
>>>>> > >
>>>>> > > wrote:
>>>>> > >
>>>>> > >> Hi, I noticed that the development on stateful functions has come
>>>>> to a
>>>>> > bit
>>>>> > >> of a halt, with a pull request to update statefun to use Flink
>>>>> 1.15
>>>>> > being
>>>>> > >> in the `open` state since May 2022.
>>>>> > >>
>>>>> > >> What do we think is the future of this sub-project?
>>>>> > >>
>>>>> > >> The background to this question is that my team is on a shared
>>>>> Flink
>>>>> > >> cluster which will soon be upgrading to Flink 1.15. If I need to
>>>>> > re-write
>>>>> > >> all our code as a native Flink job (rather than a remote stateful
>>>>> > function)
>>>>> > >> then I need to get started right away.
>>>>> > >>
>>>>> > >> Many thanks,
>>>>> > >> Fil
>>>>> > >>
>>>>> >
>>>>> > --------------------------
>>>>> > Ken Krugler
>>>>> > http://www.scaleunlimited.com
>>>>> > Custom big data solutions
>>>>> > Flink, Pinot, Solr, Elasticsearch
>>>>> >
>>>>> >
>>>>> >
>>>>> >
>>>>>
>>>>
Re: Stateful Functions with Flink 1.15 and onwards
Posted by Till Rohrmann <tr...@apache.org>.
Hi folks,
if you can open a PR for supporting Flink 1.15 Galen, then this would be
awesome. I've assigned you to this ticket. The next thing after merging
this PR would be creating a new StateFun release. Once we have merged the
PR, let's check who can help with it the fastest.
Cheers,
Till
On Mon, Oct 31, 2022 at 1:10 PM Galen Warren <ga...@cvillewarrens.com>
wrote:
> Yes, I could do that.
>
> On Mon, Oct 31, 2022 at 7:48 AM Filip Karnicki <fi...@gmail.com>
> wrote:
>
>> Hi All
>>
>> So what's the play here?
>>
>> Galen, what do you think about taking this on? Perhaps ++Till would
>> assign this jira to you (with your permission) given he's helped me out
>> with statefun work before
>> https://issues.apache.org/jira/browse/FLINK-29814
>>
>> I can try to move to move statefun to flink 1.16 when it's out
>>
>>
>> Kind regards
>> Fil
>>
>> On Thu, 27 Oct 2022 at 10:02, Filip Karnicki <fi...@gmail.com>
>> wrote:
>>
>>> Hi All
>>>
>>> Our use case is that we need to process elements for the same key
>>> sequentially, and this processing involves async operations.
>>>
>>> If any part of the processing fails, we store the offending and all
>>> subsequent incoming messages for that key in the state and not process any
>>> further messages for that key, until a retry succeeds or a human sends a
>>> 'skip' command message.
>>>
>>> diagram:
>>> https://mermaid.live/edit#pako:eNplkL1uwzAMhF-F0JQADrp76FR06tSOcQfWom3V-nFFqoUR591L20mXaqAOxHd3AC-mTZZMbfqM0wAvr00EfS62KbjYn-8C6Jui8DucTo_wmT4Oz97FEVhQqCtxXR13q_IBo-XzXWyehUc3LSu2Uyq2qIXpq2iyQ-9nmCjDSPMCmUISOuwfaEErLsVbw2272VOEDp0vmSqw5HEmC4GYsSeQpKjkv7j_buQ5tjAV4YeehOHHyQDsLAF1HbXCCyQZKB-2CTyzUOCjqUygHNBZPdxljW2MAoEaU6u0mMfGNPGqHBZJb1piasmFKlMmqxd7cqj3Dqbu0DNdfwHTGoek
>>> mermaid (in case mermaid.live goes down in the future):
>>> graph LR
>>> incoming[incoming events] --> job(Flink statefun job)
>>> commands[commands] -->|skip| job
>>> job --> |sequentially per key| remote(remote function)
>>> remote --> |on failure, delayed message to retry| remote
>>> remote --> |async puts/gets with side effects| other(other systems)
>>>
>>> Having the processing happen outside of Flink is nice-to-have from an
>>> independent scalability point of view, but is not strictly required.
>>>
>>> So long story short - no cyclic messaging, but also no way I can think
>>> of to use existing native Flink operators like async i/o (which when I last
>>> checked a few years back didn't have access to keyed state)
>>>
>>>
>>> P.S. Please note that there is already a pull request that has something
>>> to do wtih Flink 1.15, albeit without a description or a jira:
>>> https://github.com/apache/flink-statefun/pull/314
>>>
>>>
>>> On Wed, 26 Oct 2022 at 19:54, Galen Warren <ga...@cvillewarrens.com>
>>> wrote:
>>>
>>>> Hi Gordon (and others),
>>>>
>>>> I'm also using this project for stateful messaging, including messaging
>>>> among functions.
>>>>
>>>> I've contributed a small amount of code in the past and have also
>>>> enabled
>>>> Flink 1.15 compatibility in a local fork, so I might be able to help out
>>>> here.
>>>>
>>>> Thanks,
>>>> Galen
>>>>
>>>> On Wed, Oct 26, 2022 at 1:34 PM Ken Krugler <
>>>> kkrugler_lists@transpac.com>
>>>> wrote:
>>>>
>>>> > Hi Gordon,
>>>> >
>>>> > We’re using it for stateful messaging, and also calling remote
>>>> > Python-based functions.
>>>> >
>>>> > So yes, also very interested in what is going to happen with the this
>>>> > subproject in the future.
>>>> >
>>>> > — Ken
>>>> >
>>>> >
>>>> >
>>>> > > Begin forwarded message:
>>>> > >
>>>> > > From: "Tzu-Li (Gordon) Tai" <tz...@apache.org>
>>>> > > Subject: Re: Stateful Functions with Flink 1.15 and onwards
>>>> > > Date: October 26, 2022 at 10:25:26 AM PDT
>>>> > > To: dev@flink.apache.org
>>>> > > Reply-To: dev@flink.apache.org
>>>> > >
>>>> > > Hi Filip,
>>>> > >
>>>> > > Thanks for bringing this up.
>>>> > >
>>>> > > The hard truth is that committers who were previously active on the
>>>> > > StateFun subproject, including myself, all currently have other
>>>> focuses.
>>>> > > Indeed, we may need to discuss with the community on how to proceed
>>>> if
>>>> > > there seems to be no continued committer coverage.
>>>> > >
>>>> > > If it's just a matter of upgrading the supported Flink version, I'm
>>>> still
>>>> > > familiar enough with the subproject to probably be able to drive
>>>> this (or
>>>> > > if your team is up to it, I can assist you on that).
>>>> > >
>>>> > > For the long-term, as a data point I'm curious to see how many
>>>> users are
>>>> > > using StateFun in production today, and how you're using it?
>>>> > >
>>>> > > - Do your applications have arbitrary / cyclic / bi-directional
>>>> > > messaging between individual functions?
>>>> > > - Or are you utilizing StateFun simply to allow your stateful
>>>> functions
>>>> > > to run remotely as separate processes?
>>>> > >
>>>> > > If the majority is only the latter category, there might be a case
>>>> to
>>>> > > support remote functions natively in Flink (which has been a
>>>> discussion
>>>> > in
>>>> > > the past).
>>>> > >
>>>> > > Thanks,
>>>> > > Gordon
>>>> > >
>>>> > > On Wed, Oct 26, 2022 at 3:30 AM Filip Karnicki <
>>>> filip.karnicki@gmail.com
>>>> > >
>>>> > > wrote:
>>>> > >
>>>> > >> Hi, I noticed that the development on stateful functions has come
>>>> to a
>>>> > bit
>>>> > >> of a halt, with a pull request to update statefun to use Flink 1.15
>>>> > being
>>>> > >> in the `open` state since May 2022.
>>>> > >>
>>>> > >> What do we think is the future of this sub-project?
>>>> > >>
>>>> > >> The background to this question is that my team is on a shared
>>>> Flink
>>>> > >> cluster which will soon be upgrading to Flink 1.15. If I need to
>>>> > re-write
>>>> > >> all our code as a native Flink job (rather than a remote stateful
>>>> > function)
>>>> > >> then I need to get started right away.
>>>> > >>
>>>> > >> Many thanks,
>>>> > >> Fil
>>>> > >>
>>>> >
>>>> > --------------------------
>>>> > Ken Krugler
>>>> > http://www.scaleunlimited.com
>>>> > Custom big data solutions
>>>> > Flink, Pinot, Solr, Elasticsearch
>>>> >
>>>> >
>>>> >
>>>> >
>>>>
>>>
Re: Stateful Functions with Flink 1.15 and onwards
Posted by Galen Warren <ga...@cvillewarrens.com>.
Yes, I could do that.
On Mon, Oct 31, 2022 at 7:48 AM Filip Karnicki <fi...@gmail.com>
wrote:
> Hi All
>
> So what's the play here?
>
> Galen, what do you think about taking this on? Perhaps ++Till would assign
> this jira to you (with your permission) given he's helped me out
> with statefun work before
> https://issues.apache.org/jira/browse/FLINK-29814
>
> I can try to move to move statefun to flink 1.16 when it's out
>
>
> Kind regards
> Fil
>
> On Thu, 27 Oct 2022 at 10:02, Filip Karnicki <fi...@gmail.com>
> wrote:
>
>> Hi All
>>
>> Our use case is that we need to process elements for the same key
>> sequentially, and this processing involves async operations.
>>
>> If any part of the processing fails, we store the offending and all
>> subsequent incoming messages for that key in the state and not process any
>> further messages for that key, until a retry succeeds or a human sends a
>> 'skip' command message.
>>
>> diagram:
>> https://mermaid.live/edit#pako:eNplkL1uwzAMhF-F0JQADrp76FR06tSOcQfWom3V-nFFqoUR591L20mXaqAOxHd3AC-mTZZMbfqM0wAvr00EfS62KbjYn-8C6Jui8DucTo_wmT4Oz97FEVhQqCtxXR13q_IBo-XzXWyehUc3LSu2Uyq2qIXpq2iyQ-9nmCjDSPMCmUISOuwfaEErLsVbw2272VOEDp0vmSqw5HEmC4GYsSeQpKjkv7j_buQ5tjAV4YeehOHHyQDsLAF1HbXCCyQZKB-2CTyzUOCjqUygHNBZPdxljW2MAoEaU6u0mMfGNPGqHBZJb1piasmFKlMmqxd7cqj3Dqbu0DNdfwHTGoek
>> mermaid (in case mermaid.live goes down in the future):
>> graph LR
>> incoming[incoming events] --> job(Flink statefun job)
>> commands[commands] -->|skip| job
>> job --> |sequentially per key| remote(remote function)
>> remote --> |on failure, delayed message to retry| remote
>> remote --> |async puts/gets with side effects| other(other systems)
>>
>> Having the processing happen outside of Flink is nice-to-have from an
>> independent scalability point of view, but is not strictly required.
>>
>> So long story short - no cyclic messaging, but also no way I can think of
>> to use existing native Flink operators like async i/o (which when I last
>> checked a few years back didn't have access to keyed state)
>>
>>
>> P.S. Please note that there is already a pull request that has something
>> to do wtih Flink 1.15, albeit without a description or a jira:
>> https://github.com/apache/flink-statefun/pull/314
>>
>>
>> On Wed, 26 Oct 2022 at 19:54, Galen Warren <ga...@cvillewarrens.com>
>> wrote:
>>
>>> Hi Gordon (and others),
>>>
>>> I'm also using this project for stateful messaging, including messaging
>>> among functions.
>>>
>>> I've contributed a small amount of code in the past and have also enabled
>>> Flink 1.15 compatibility in a local fork, so I might be able to help out
>>> here.
>>>
>>> Thanks,
>>> Galen
>>>
>>> On Wed, Oct 26, 2022 at 1:34 PM Ken Krugler <kkrugler_lists@transpac.com
>>> >
>>> wrote:
>>>
>>> > Hi Gordon,
>>> >
>>> > We’re using it for stateful messaging, and also calling remote
>>> > Python-based functions.
>>> >
>>> > So yes, also very interested in what is going to happen with the this
>>> > subproject in the future.
>>> >
>>> > — Ken
>>> >
>>> >
>>> >
>>> > > Begin forwarded message:
>>> > >
>>> > > From: "Tzu-Li (Gordon) Tai" <tz...@apache.org>
>>> > > Subject: Re: Stateful Functions with Flink 1.15 and onwards
>>> > > Date: October 26, 2022 at 10:25:26 AM PDT
>>> > > To: dev@flink.apache.org
>>> > > Reply-To: dev@flink.apache.org
>>> > >
>>> > > Hi Filip,
>>> > >
>>> > > Thanks for bringing this up.
>>> > >
>>> > > The hard truth is that committers who were previously active on the
>>> > > StateFun subproject, including myself, all currently have other
>>> focuses.
>>> > > Indeed, we may need to discuss with the community on how to proceed
>>> if
>>> > > there seems to be no continued committer coverage.
>>> > >
>>> > > If it's just a matter of upgrading the supported Flink version, I'm
>>> still
>>> > > familiar enough with the subproject to probably be able to drive
>>> this (or
>>> > > if your team is up to it, I can assist you on that).
>>> > >
>>> > > For the long-term, as a data point I'm curious to see how many users
>>> are
>>> > > using StateFun in production today, and how you're using it?
>>> > >
>>> > > - Do your applications have arbitrary / cyclic / bi-directional
>>> > > messaging between individual functions?
>>> > > - Or are you utilizing StateFun simply to allow your stateful
>>> functions
>>> > > to run remotely as separate processes?
>>> > >
>>> > > If the majority is only the latter category, there might be a case to
>>> > > support remote functions natively in Flink (which has been a
>>> discussion
>>> > in
>>> > > the past).
>>> > >
>>> > > Thanks,
>>> > > Gordon
>>> > >
>>> > > On Wed, Oct 26, 2022 at 3:30 AM Filip Karnicki <
>>> filip.karnicki@gmail.com
>>> > >
>>> > > wrote:
>>> > >
>>> > >> Hi, I noticed that the development on stateful functions has come
>>> to a
>>> > bit
>>> > >> of a halt, with a pull request to update statefun to use Flink 1.15
>>> > being
>>> > >> in the `open` state since May 2022.
>>> > >>
>>> > >> What do we think is the future of this sub-project?
>>> > >>
>>> > >> The background to this question is that my team is on a shared Flink
>>> > >> cluster which will soon be upgrading to Flink 1.15. If I need to
>>> > re-write
>>> > >> all our code as a native Flink job (rather than a remote stateful
>>> > function)
>>> > >> then I need to get started right away.
>>> > >>
>>> > >> Many thanks,
>>> > >> Fil
>>> > >>
>>> >
>>> > --------------------------
>>> > Ken Krugler
>>> > http://www.scaleunlimited.com
>>> > Custom big data solutions
>>> > Flink, Pinot, Solr, Elasticsearch
>>> >
>>> >
>>> >
>>> >
>>>
>>
Re: Stateful Functions with Flink 1.15 and onwards
Posted by Filip Karnicki <fi...@gmail.com>.
Hi All
So what's the play here?
Galen, what do you think about taking this on? Perhaps ++Till would assign
this jira to you (with your permission) given he's helped me out
with statefun work before
https://issues.apache.org/jira/browse/FLINK-29814
I can try to move to move statefun to flink 1.16 when it's out
Kind regards
Fil
On Thu, 27 Oct 2022 at 10:02, Filip Karnicki <fi...@gmail.com>
wrote:
> Hi All
>
> Our use case is that we need to process elements for the same key
> sequentially, and this processing involves async operations.
>
> If any part of the processing fails, we store the offending and all
> subsequent incoming messages for that key in the state and not process any
> further messages for that key, until a retry succeeds or a human sends a
> 'skip' command message.
>
> diagram:
> https://mermaid.live/edit#pako:eNplkL1uwzAMhF-F0JQADrp76FR06tSOcQfWom3V-nFFqoUR591L20mXaqAOxHd3AC-mTZZMbfqM0wAvr00EfS62KbjYn-8C6Jui8DucTo_wmT4Oz97FEVhQqCtxXR13q_IBo-XzXWyehUc3LSu2Uyq2qIXpq2iyQ-9nmCjDSPMCmUISOuwfaEErLsVbw2272VOEDp0vmSqw5HEmC4GYsSeQpKjkv7j_buQ5tjAV4YeehOHHyQDsLAF1HbXCCyQZKB-2CTyzUOCjqUygHNBZPdxljW2MAoEaU6u0mMfGNPGqHBZJb1piasmFKlMmqxd7cqj3Dqbu0DNdfwHTGoek
> mermaid (in case mermaid.live goes down in the future):
> graph LR
> incoming[incoming events] --> job(Flink statefun job)
> commands[commands] -->|skip| job
> job --> |sequentially per key| remote(remote function)
> remote --> |on failure, delayed message to retry| remote
> remote --> |async puts/gets with side effects| other(other systems)
>
> Having the processing happen outside of Flink is nice-to-have from an
> independent scalability point of view, but is not strictly required.
>
> So long story short - no cyclic messaging, but also no way I can think of
> to use existing native Flink operators like async i/o (which when I last
> checked a few years back didn't have access to keyed state)
>
>
> P.S. Please note that there is already a pull request that has something
> to do wtih Flink 1.15, albeit without a description or a jira:
> https://github.com/apache/flink-statefun/pull/314
>
>
> On Wed, 26 Oct 2022 at 19:54, Galen Warren <ga...@cvillewarrens.com>
> wrote:
>
>> Hi Gordon (and others),
>>
>> I'm also using this project for stateful messaging, including messaging
>> among functions.
>>
>> I've contributed a small amount of code in the past and have also enabled
>> Flink 1.15 compatibility in a local fork, so I might be able to help out
>> here.
>>
>> Thanks,
>> Galen
>>
>> On Wed, Oct 26, 2022 at 1:34 PM Ken Krugler <kk...@transpac.com>
>> wrote:
>>
>> > Hi Gordon,
>> >
>> > We’re using it for stateful messaging, and also calling remote
>> > Python-based functions.
>> >
>> > So yes, also very interested in what is going to happen with the this
>> > subproject in the future.
>> >
>> > — Ken
>> >
>> >
>> >
>> > > Begin forwarded message:
>> > >
>> > > From: "Tzu-Li (Gordon) Tai" <tz...@apache.org>
>> > > Subject: Re: Stateful Functions with Flink 1.15 and onwards
>> > > Date: October 26, 2022 at 10:25:26 AM PDT
>> > > To: dev@flink.apache.org
>> > > Reply-To: dev@flink.apache.org
>> > >
>> > > Hi Filip,
>> > >
>> > > Thanks for bringing this up.
>> > >
>> > > The hard truth is that committers who were previously active on the
>> > > StateFun subproject, including myself, all currently have other
>> focuses.
>> > > Indeed, we may need to discuss with the community on how to proceed if
>> > > there seems to be no continued committer coverage.
>> > >
>> > > If it's just a matter of upgrading the supported Flink version, I'm
>> still
>> > > familiar enough with the subproject to probably be able to drive this
>> (or
>> > > if your team is up to it, I can assist you on that).
>> > >
>> > > For the long-term, as a data point I'm curious to see how many users
>> are
>> > > using StateFun in production today, and how you're using it?
>> > >
>> > > - Do your applications have arbitrary / cyclic / bi-directional
>> > > messaging between individual functions?
>> > > - Or are you utilizing StateFun simply to allow your stateful
>> functions
>> > > to run remotely as separate processes?
>> > >
>> > > If the majority is only the latter category, there might be a case to
>> > > support remote functions natively in Flink (which has been a
>> discussion
>> > in
>> > > the past).
>> > >
>> > > Thanks,
>> > > Gordon
>> > >
>> > > On Wed, Oct 26, 2022 at 3:30 AM Filip Karnicki <
>> filip.karnicki@gmail.com
>> > >
>> > > wrote:
>> > >
>> > >> Hi, I noticed that the development on stateful functions has come to
>> a
>> > bit
>> > >> of a halt, with a pull request to update statefun to use Flink 1.15
>> > being
>> > >> in the `open` state since May 2022.
>> > >>
>> > >> What do we think is the future of this sub-project?
>> > >>
>> > >> The background to this question is that my team is on a shared Flink
>> > >> cluster which will soon be upgrading to Flink 1.15. If I need to
>> > re-write
>> > >> all our code as a native Flink job (rather than a remote stateful
>> > function)
>> > >> then I need to get started right away.
>> > >>
>> > >> Many thanks,
>> > >> Fil
>> > >>
>> >
>> > --------------------------
>> > Ken Krugler
>> > http://www.scaleunlimited.com
>> > Custom big data solutions
>> > Flink, Pinot, Solr, Elasticsearch
>> >
>> >
>> >
>> >
>>
>
Re: Stateful Functions with Flink 1.15 and onwards
Posted by Filip Karnicki <fi...@gmail.com>.
Hi All
Our use case is that we need to process elements for the same key
sequentially, and this processing involves async operations.
If any part of the processing fails, we store the offending and all
subsequent incoming messages for that key in the state and not process any
further messages for that key, until a retry succeeds or a human sends a
'skip' command message.
diagram:
https://mermaid.live/edit#pako:eNplkL1uwzAMhF-F0JQADrp76FR06tSOcQfWom3V-nFFqoUR591L20mXaqAOxHd3AC-mTZZMbfqM0wAvr00EfS62KbjYn-8C6Jui8DucTo_wmT4Oz97FEVhQqCtxXR13q_IBo-XzXWyehUc3LSu2Uyq2qIXpq2iyQ-9nmCjDSPMCmUISOuwfaEErLsVbw2272VOEDp0vmSqw5HEmC4GYsSeQpKjkv7j_buQ5tjAV4YeehOHHyQDsLAF1HbXCCyQZKB-2CTyzUOCjqUygHNBZPdxljW2MAoEaU6u0mMfGNPGqHBZJb1piasmFKlMmqxd7cqj3Dqbu0DNdfwHTGoek
mermaid (in case mermaid.live goes down in the future):
graph LR
incoming[incoming events] --> job(Flink statefun job)
commands[commands] -->|skip| job
job --> |sequentially per key| remote(remote function)
remote --> |on failure, delayed message to retry| remote
remote --> |async puts/gets with side effects| other(other systems)
Having the processing happen outside of Flink is nice-to-have from an
independent scalability point of view, but is not strictly required.
So long story short - no cyclic messaging, but also no way I can think of
to use existing native Flink operators like async i/o (which when I last
checked a few years back didn't have access to keyed state)
P.S. Please note that there is already a pull request that has something to
do wtih Flink 1.15, albeit without a description or a jira:
https://github.com/apache/flink-statefun/pull/314
On Wed, 26 Oct 2022 at 19:54, Galen Warren <ga...@cvillewarrens.com> wrote:
> Hi Gordon (and others),
>
> I'm also using this project for stateful messaging, including messaging
> among functions.
>
> I've contributed a small amount of code in the past and have also enabled
> Flink 1.15 compatibility in a local fork, so I might be able to help out
> here.
>
> Thanks,
> Galen
>
> On Wed, Oct 26, 2022 at 1:34 PM Ken Krugler <kk...@transpac.com>
> wrote:
>
> > Hi Gordon,
> >
> > We’re using it for stateful messaging, and also calling remote
> > Python-based functions.
> >
> > So yes, also very interested in what is going to happen with the this
> > subproject in the future.
> >
> > — Ken
> >
> >
> >
> > > Begin forwarded message:
> > >
> > > From: "Tzu-Li (Gordon) Tai" <tz...@apache.org>
> > > Subject: Re: Stateful Functions with Flink 1.15 and onwards
> > > Date: October 26, 2022 at 10:25:26 AM PDT
> > > To: dev@flink.apache.org
> > > Reply-To: dev@flink.apache.org
> > >
> > > Hi Filip,
> > >
> > > Thanks for bringing this up.
> > >
> > > The hard truth is that committers who were previously active on the
> > > StateFun subproject, including myself, all currently have other
> focuses.
> > > Indeed, we may need to discuss with the community on how to proceed if
> > > there seems to be no continued committer coverage.
> > >
> > > If it's just a matter of upgrading the supported Flink version, I'm
> still
> > > familiar enough with the subproject to probably be able to drive this
> (or
> > > if your team is up to it, I can assist you on that).
> > >
> > > For the long-term, as a data point I'm curious to see how many users
> are
> > > using StateFun in production today, and how you're using it?
> > >
> > > - Do your applications have arbitrary / cyclic / bi-directional
> > > messaging between individual functions?
> > > - Or are you utilizing StateFun simply to allow your stateful
> functions
> > > to run remotely as separate processes?
> > >
> > > If the majority is only the latter category, there might be a case to
> > > support remote functions natively in Flink (which has been a discussion
> > in
> > > the past).
> > >
> > > Thanks,
> > > Gordon
> > >
> > > On Wed, Oct 26, 2022 at 3:30 AM Filip Karnicki <
> filip.karnicki@gmail.com
> > >
> > > wrote:
> > >
> > >> Hi, I noticed that the development on stateful functions has come to a
> > bit
> > >> of a halt, with a pull request to update statefun to use Flink 1.15
> > being
> > >> in the `open` state since May 2022.
> > >>
> > >> What do we think is the future of this sub-project?
> > >>
> > >> The background to this question is that my team is on a shared Flink
> > >> cluster which will soon be upgrading to Flink 1.15. If I need to
> > re-write
> > >> all our code as a native Flink job (rather than a remote stateful
> > function)
> > >> then I need to get started right away.
> > >>
> > >> Many thanks,
> > >> Fil
> > >>
> >
> > --------------------------
> > Ken Krugler
> > http://www.scaleunlimited.com
> > Custom big data solutions
> > Flink, Pinot, Solr, Elasticsearch
> >
> >
> >
> >
>
Re: Stateful Functions with Flink 1.15 and onwards
Posted by Galen Warren <ga...@cvillewarrens.com>.
Hi Gordon (and others),
I'm also using this project for stateful messaging, including messaging
among functions.
I've contributed a small amount of code in the past and have also enabled
Flink 1.15 compatibility in a local fork, so I might be able to help out
here.
Thanks,
Galen
On Wed, Oct 26, 2022 at 1:34 PM Ken Krugler <kk...@transpac.com>
wrote:
> Hi Gordon,
>
> We’re using it for stateful messaging, and also calling remote
> Python-based functions.
>
> So yes, also very interested in what is going to happen with the this
> subproject in the future.
>
> — Ken
>
>
>
> > Begin forwarded message:
> >
> > From: "Tzu-Li (Gordon) Tai" <tz...@apache.org>
> > Subject: Re: Stateful Functions with Flink 1.15 and onwards
> > Date: October 26, 2022 at 10:25:26 AM PDT
> > To: dev@flink.apache.org
> > Reply-To: dev@flink.apache.org
> >
> > Hi Filip,
> >
> > Thanks for bringing this up.
> >
> > The hard truth is that committers who were previously active on the
> > StateFun subproject, including myself, all currently have other focuses.
> > Indeed, we may need to discuss with the community on how to proceed if
> > there seems to be no continued committer coverage.
> >
> > If it's just a matter of upgrading the supported Flink version, I'm still
> > familiar enough with the subproject to probably be able to drive this (or
> > if your team is up to it, I can assist you on that).
> >
> > For the long-term, as a data point I'm curious to see how many users are
> > using StateFun in production today, and how you're using it?
> >
> > - Do your applications have arbitrary / cyclic / bi-directional
> > messaging between individual functions?
> > - Or are you utilizing StateFun simply to allow your stateful functions
> > to run remotely as separate processes?
> >
> > If the majority is only the latter category, there might be a case to
> > support remote functions natively in Flink (which has been a discussion
> in
> > the past).
> >
> > Thanks,
> > Gordon
> >
> > On Wed, Oct 26, 2022 at 3:30 AM Filip Karnicki <filip.karnicki@gmail.com
> >
> > wrote:
> >
> >> Hi, I noticed that the development on stateful functions has come to a
> bit
> >> of a halt, with a pull request to update statefun to use Flink 1.15
> being
> >> in the `open` state since May 2022.
> >>
> >> What do we think is the future of this sub-project?
> >>
> >> The background to this question is that my team is on a shared Flink
> >> cluster which will soon be upgrading to Flink 1.15. If I need to
> re-write
> >> all our code as a native Flink job (rather than a remote stateful
> function)
> >> then I need to get started right away.
> >>
> >> Many thanks,
> >> Fil
> >>
>
> --------------------------
> Ken Krugler
> http://www.scaleunlimited.com
> Custom big data solutions
> Flink, Pinot, Solr, Elasticsearch
>
>
>
>
Fwd: Stateful Functions with Flink 1.15 and onwards
Posted by Ken Krugler <kk...@transpac.com>.
Hi Gordon,
We’re using it for stateful messaging, and also calling remote Python-based functions.
So yes, also very interested in what is going to happen with the this subproject in the future.
— Ken
> Begin forwarded message:
>
> From: "Tzu-Li (Gordon) Tai" <tz...@apache.org>
> Subject: Re: Stateful Functions with Flink 1.15 and onwards
> Date: October 26, 2022 at 10:25:26 AM PDT
> To: dev@flink.apache.org
> Reply-To: dev@flink.apache.org
>
> Hi Filip,
>
> Thanks for bringing this up.
>
> The hard truth is that committers who were previously active on the
> StateFun subproject, including myself, all currently have other focuses.
> Indeed, we may need to discuss with the community on how to proceed if
> there seems to be no continued committer coverage.
>
> If it's just a matter of upgrading the supported Flink version, I'm still
> familiar enough with the subproject to probably be able to drive this (or
> if your team is up to it, I can assist you on that).
>
> For the long-term, as a data point I'm curious to see how many users are
> using StateFun in production today, and how you're using it?
>
> - Do your applications have arbitrary / cyclic / bi-directional
> messaging between individual functions?
> - Or are you utilizing StateFun simply to allow your stateful functions
> to run remotely as separate processes?
>
> If the majority is only the latter category, there might be a case to
> support remote functions natively in Flink (which has been a discussion in
> the past).
>
> Thanks,
> Gordon
>
> On Wed, Oct 26, 2022 at 3:30 AM Filip Karnicki <fi...@gmail.com>
> wrote:
>
>> Hi, I noticed that the development on stateful functions has come to a bit
>> of a halt, with a pull request to update statefun to use Flink 1.15 being
>> in the `open` state since May 2022.
>>
>> What do we think is the future of this sub-project?
>>
>> The background to this question is that my team is on a shared Flink
>> cluster which will soon be upgrading to Flink 1.15. If I need to re-write
>> all our code as a native Flink job (rather than a remote stateful function)
>> then I need to get started right away.
>>
>> Many thanks,
>> Fil
>>
--------------------------
Ken Krugler
http://www.scaleunlimited.com
Custom big data solutions
Flink, Pinot, Solr, Elasticsearch
Re: Stateful Functions with Flink 1.15 and onwards
Posted by "Tzu-Li (Gordon) Tai" <tz...@apache.org>.
Hi Filip,
Thanks for bringing this up.
The hard truth is that committers who were previously active on the
StateFun subproject, including myself, all currently have other focuses.
Indeed, we may need to discuss with the community on how to proceed if
there seems to be no continued committer coverage.
If it's just a matter of upgrading the supported Flink version, I'm still
familiar enough with the subproject to probably be able to drive this (or
if your team is up to it, I can assist you on that).
For the long-term, as a data point I'm curious to see how many users are
using StateFun in production today, and how you're using it?
- Do your applications have arbitrary / cyclic / bi-directional
messaging between individual functions?
- Or are you utilizing StateFun simply to allow your stateful functions
to run remotely as separate processes?
If the majority is only the latter category, there might be a case to
support remote functions natively in Flink (which has been a discussion in
the past).
Thanks,
Gordon
On Wed, Oct 26, 2022 at 3:30 AM Filip Karnicki <fi...@gmail.com>
wrote:
> Hi, I noticed that the development on stateful functions has come to a bit
> of a halt, with a pull request to update statefun to use Flink 1.15 being
> in the `open` state since May 2022.
>
> What do we think is the future of this sub-project?
>
> The background to this question is that my team is on a shared Flink
> cluster which will soon be upgrading to Flink 1.15. If I need to re-write
> all our code as a native Flink job (rather than a remote stateful function)
> then I need to get started right away.
>
> Many thanks,
> Fil
>