You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Timothy Bess <td...@gmail.com> on 2020/09/12 05:26:10 UTC
Flink Stateful Functions API
The flink stateful function Python API looks cool, but is there a
documented spec for how it communicates with Flink? I'd like to implement
an SDK in Haskell if I can.
Re: Flink Stateful Functions API
Posted by Timothy Bess <td...@gmail.com>.
Hi Gordon,
Awesome thanks for pointing me to that. I'll polish it up a bit and write
some tests and submit a PR for that page then. Great job on all this,
really love what you guys are doing. Keep up the good work!
Thanks,
Tim
On Tue, Sep 15, 2020 at 4:33 AM Tzu-Li (Gordon) Tai <tz...@apache.org>
wrote:
> Hi,
>
> Just wanted to add that we recently added a page in the docs to list 3rd
> party SDKs:
> https://ci.apache.org/projects/flink/flink-statefun-docs-master/sdk/external.html
> .
> Once you're ready with your Haskell and would want it to be listed on that
> page, feel free to open a pull request for that :)
> You can find the source for the documentation content here:
> https://github.com/apache/flink-statefun/blob/master/docs/sdk/external.md
>
> Cheers,
> Gordon
>
> On Tue, Sep 15, 2020 at 2:21 AM Timothy Bess <td...@gmail.com> wrote:
>
>> Hi Igal,
>>
>> Thanks! It wasn't too hard to figure out, the structure makes sense. Will
>> be interesting to see how it evolves once you start lazily fetching state
>> and add other features.
>>
>> Yeah I expected there may not be many people to review it for mistakes
>> haha, but luckily it's only ~ 245 lines of Haskell so it shouldn't be
>> terribly difficult to maintain. I'm definitely planning to maintain this
>> and will probably start using it at work once we get around to the stateful
>> side of our architecture. I'll add a note to the README that the API is
>> still in flux while I iron out a few things and write a test suite for it.
>> Would definitely be exciting to see Haskell on the list of 3rd party SDKs
>> though! Glad to see Flink moving in this direction, always have been a huge
>> fan of the functionality it provides, but wanted a bit more decoupling from
>> the Flink streaming APIs themselves. This project seems, to me, like
>> someone has essentially figured out how to do polyglot Akka/OTP at large
>> scale with internal ACID state (which is my dream architecture).
>>
>> Thanks,
>>
>> Tim
>>
>> On Mon, Sep 14, 2020 at 1:09 PM Igal Shilman <ig...@ververica.com> wrote:
>>
>>> Hi Tim,
>>>
>>> This is amazing! I hope that it wasn't too much trouble for you to
>>> figure out the bits that needed to be implemented for the RequestReply
>>> protocol.
>>>
>>> Unfortunately I don't know enough Haskell to provide you any
>>> meaningful feedback for the SDK itself, not to mention to maintain it at
>>> this point.
>>> But what I can suggest is that if you are interested to continue
>>> maintaining and evolving this, we can add this to a list of 3rd party SDKs
>>> that are developed
>>> by the community. And if eventually enough interest would arise and you
>>> would like to contribute this then we can revisit that!
>>>
>>> In any case, this is incredible and I'm looking forward to seeing
>>> exciting new use cases with Haskell and StateFun :-)
>>>
>>> Thanks,
>>> Igal.
>>>
>>>
>>>
>>>
>>> On Mon, Sep 14, 2020 at 3:01 PM Timothy Bess <td...@gmail.com> wrote:
>>>
>>>> Hi Gordon and Dawid,
>>>>
>>>> Thanks for the responses! I actually ended up just biting the bullet
>>>> and digging through the Python implementation to figure it out after I
>>>> wrote that email. Was more worried about accidentally depending on
>>>> something internal to the Python API that may not be "spec", but it seemed
>>>> pretty straightforward when I read through it actually. I made a small
>>>> implementation over the weekend and got the greeter example working. Not
>>>> sure how familiar anyone is here with Haskell, but here's the code
>>>> <https://github.com/tdbgamer/flink-statefulfun-hs> if you're
>>>> interested. The example is pretty simple, just have a function that takes
>>>> messages and sends them out to a counter ID'd by the user's name.
>>>>
>>>> But essentially you interact with the Flink SDK from this one typeclass:
>>>> [image: image.png]
>>>>
>>>>
>>>> And it essentially just runs through the batches in memory, takes the
>>>> resulting state, converts it to a "FromFunction" and ships it all back.
>>>> Haskell has the same sort of green threading concurrency model as Erlang/Go
>>>> if you're familiar with that, so it was actually very fast when I was
>>>> testing it.
>>>>
>>>> With these sort of SDKs, do people generally get it merged into some
>>>> Apache/Flink repo and become official? Or does it just stay on my Github?
>>>>
>>>> Thanks,
>>>>
>>>> Tim
>>>>
>>>> On Mon, Sep 14, 2020 at 4:55 AM Tzu-Li (Gordon) Tai <
>>>> tzulitai@apache.org> wrote:
>>>>
>>>>> Hi!
>>>>>
>>>>> Dawid is right, there currently is no developer documentation for the
>>>>> remote request-reply protocol.
>>>>> One reason for this is that the protocol isn't considered a fully
>>>>> stable user-facing interface yet, and thus not yet properly advertised in
>>>>> the documentation.
>>>>> However, there are plans to revisit it and announce it as publicly
>>>>> stable in the near future releases.
>>>>>
>>>>> In the meantime, the Python SDK and Aljoscha's Rust SDK are good
>>>>> reference examples of implementation of the protocol across different
>>>>> languages.
>>>>> The request body from Flink and expected response body from functions
>>>>> are essentially these [1] Protobuf messages, ToFunction (request) and
>>>>> FromFunction (response).
>>>>>
>>>>> If you bump into any issues when implementing, please feel free to let
>>>>> us know on the mailing lists as well.
>>>>> A Haskell SDK is definitely interesting to see implemented :)
>>>>>
>>>>> Cheers,
>>>>> Gordon
>>>>>
>>>>> [1]
>>>>> https://github.com/apache/flink-statefun/blob/master/statefun-flink/statefun-flink-core/src/main/protobuf/http-function.proto
>>>>>
>>>>> On Mon, Sep 14, 2020 at 3:46 PM Dawid Wysakowicz <
>>>>> dwysakowicz@apache.org> wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> Not sure if there is a "developer" documentation for the protocol. I
>>>>>> am
>>>>>> cc'ing Igal and Gordon who know better than I if there is one.
>>>>>>
>>>>>> To give you some hints though. If I am correct the Python API is
>>>>>> implemented as a so called remote functions [1][2], which communicate
>>>>>> with Flink via HTTP/gRPC. Besides the bundled Python API you can also
>>>>>> use a Rust SDK[3] implemented by my colleague Aljoscha as a reference.
>>>>>>
>>>>>> BTW, it would be really cool to see an SDK written in Haskell ;)
>>>>>>
>>>>>> Best,
>>>>>>
>>>>>> Dawid
>>>>>>
>>>>>> [1]
>>>>>>
>>>>>> https://ci.apache.org/projects/flink/flink-statefun-docs-release-2.1/concepts/distributed_architecture.html#remote-functions
>>>>>>
>>>>>> [2]
>>>>>>
>>>>>> https://ci.apache.org/projects/flink/flink-statefun-docs-release-2.1/sdk/modules.html#remote-module
>>>>>>
>>>>>> [3] https://github.com/aljoscha/statefun-rust
>>>>>>
>>>>>> On 12/09/2020 07:26, Timothy Bess wrote:
>>>>>> > The flink stateful function Python API looks cool, but is there a
>>>>>> > documented spec for how it communicates with Flink? I'd like to
>>>>>> > implement an SDK in Haskell if I can.
>>>>>>
>>>>>>
Re: Flink Stateful Functions API
Posted by "Tzu-Li (Gordon) Tai" <tz...@apache.org>.
Hi,
Just wanted to add that we recently added a page in the docs to list 3rd
party SDKs:
https://ci.apache.org/projects/flink/flink-statefun-docs-master/sdk/external.html
.
Once you're ready with your Haskell and would want it to be listed on that
page, feel free to open a pull request for that :)
You can find the source for the documentation content here:
https://github.com/apache/flink-statefun/blob/master/docs/sdk/external.md
Cheers,
Gordon
On Tue, Sep 15, 2020 at 2:21 AM Timothy Bess <td...@gmail.com> wrote:
> Hi Igal,
>
> Thanks! It wasn't too hard to figure out, the structure makes sense. Will
> be interesting to see how it evolves once you start lazily fetching state
> and add other features.
>
> Yeah I expected there may not be many people to review it for mistakes
> haha, but luckily it's only ~ 245 lines of Haskell so it shouldn't be
> terribly difficult to maintain. I'm definitely planning to maintain this
> and will probably start using it at work once we get around to the stateful
> side of our architecture. I'll add a note to the README that the API is
> still in flux while I iron out a few things and write a test suite for it.
> Would definitely be exciting to see Haskell on the list of 3rd party SDKs
> though! Glad to see Flink moving in this direction, always have been a huge
> fan of the functionality it provides, but wanted a bit more decoupling from
> the Flink streaming APIs themselves. This project seems, to me, like
> someone has essentially figured out how to do polyglot Akka/OTP at large
> scale with internal ACID state (which is my dream architecture).
>
> Thanks,
>
> Tim
>
> On Mon, Sep 14, 2020 at 1:09 PM Igal Shilman <ig...@ververica.com> wrote:
>
>> Hi Tim,
>>
>> This is amazing! I hope that it wasn't too much trouble for you to figure
>> out the bits that needed to be implemented for the RequestReply protocol.
>>
>> Unfortunately I don't know enough Haskell to provide you any
>> meaningful feedback for the SDK itself, not to mention to maintain it at
>> this point.
>> But what I can suggest is that if you are interested to continue
>> maintaining and evolving this, we can add this to a list of 3rd party SDKs
>> that are developed
>> by the community. And if eventually enough interest would arise and you
>> would like to contribute this then we can revisit that!
>>
>> In any case, this is incredible and I'm looking forward to seeing
>> exciting new use cases with Haskell and StateFun :-)
>>
>> Thanks,
>> Igal.
>>
>>
>>
>>
>> On Mon, Sep 14, 2020 at 3:01 PM Timothy Bess <td...@gmail.com> wrote:
>>
>>> Hi Gordon and Dawid,
>>>
>>> Thanks for the responses! I actually ended up just biting the bullet and
>>> digging through the Python implementation to figure it out after I wrote
>>> that email. Was more worried about accidentally depending on something
>>> internal to the Python API that may not be "spec", but it seemed pretty
>>> straightforward when I read through it actually. I made a small
>>> implementation over the weekend and got the greeter example working. Not
>>> sure how familiar anyone is here with Haskell, but here's the code
>>> <https://github.com/tdbgamer/flink-statefulfun-hs> if you're
>>> interested. The example is pretty simple, just have a function that takes
>>> messages and sends them out to a counter ID'd by the user's name.
>>>
>>> But essentially you interact with the Flink SDK from this one typeclass:
>>> [image: image.png]
>>>
>>>
>>> And it essentially just runs through the batches in memory, takes the
>>> resulting state, converts it to a "FromFunction" and ships it all back.
>>> Haskell has the same sort of green threading concurrency model as Erlang/Go
>>> if you're familiar with that, so it was actually very fast when I was
>>> testing it.
>>>
>>> With these sort of SDKs, do people generally get it merged into some
>>> Apache/Flink repo and become official? Or does it just stay on my Github?
>>>
>>> Thanks,
>>>
>>> Tim
>>>
>>> On Mon, Sep 14, 2020 at 4:55 AM Tzu-Li (Gordon) Tai <tz...@apache.org>
>>> wrote:
>>>
>>>> Hi!
>>>>
>>>> Dawid is right, there currently is no developer documentation for the
>>>> remote request-reply protocol.
>>>> One reason for this is that the protocol isn't considered a fully
>>>> stable user-facing interface yet, and thus not yet properly advertised in
>>>> the documentation.
>>>> However, there are plans to revisit it and announce it as publicly
>>>> stable in the near future releases.
>>>>
>>>> In the meantime, the Python SDK and Aljoscha's Rust SDK are good
>>>> reference examples of implementation of the protocol across different
>>>> languages.
>>>> The request body from Flink and expected response body from functions
>>>> are essentially these [1] Protobuf messages, ToFunction (request) and
>>>> FromFunction (response).
>>>>
>>>> If you bump into any issues when implementing, please feel free to let
>>>> us know on the mailing lists as well.
>>>> A Haskell SDK is definitely interesting to see implemented :)
>>>>
>>>> Cheers,
>>>> Gordon
>>>>
>>>> [1]
>>>> https://github.com/apache/flink-statefun/blob/master/statefun-flink/statefun-flink-core/src/main/protobuf/http-function.proto
>>>>
>>>> On Mon, Sep 14, 2020 at 3:46 PM Dawid Wysakowicz <
>>>> dwysakowicz@apache.org> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> Not sure if there is a "developer" documentation for the protocol. I am
>>>>> cc'ing Igal and Gordon who know better than I if there is one.
>>>>>
>>>>> To give you some hints though. If I am correct the Python API is
>>>>> implemented as a so called remote functions [1][2], which communicate
>>>>> with Flink via HTTP/gRPC. Besides the bundled Python API you can also
>>>>> use a Rust SDK[3] implemented by my colleague Aljoscha as a reference.
>>>>>
>>>>> BTW, it would be really cool to see an SDK written in Haskell ;)
>>>>>
>>>>> Best,
>>>>>
>>>>> Dawid
>>>>>
>>>>> [1]
>>>>>
>>>>> https://ci.apache.org/projects/flink/flink-statefun-docs-release-2.1/concepts/distributed_architecture.html#remote-functions
>>>>>
>>>>> [2]
>>>>>
>>>>> https://ci.apache.org/projects/flink/flink-statefun-docs-release-2.1/sdk/modules.html#remote-module
>>>>>
>>>>> [3] https://github.com/aljoscha/statefun-rust
>>>>>
>>>>> On 12/09/2020 07:26, Timothy Bess wrote:
>>>>> > The flink stateful function Python API looks cool, but is there a
>>>>> > documented spec for how it communicates with Flink? I'd like to
>>>>> > implement an SDK in Haskell if I can.
>>>>>
>>>>>
Re: Flink Stateful Functions API
Posted by Timothy Bess <td...@gmail.com>.
Hi Igal,
Thanks! It wasn't too hard to figure out, the structure makes sense. Will
be interesting to see how it evolves once you start lazily fetching state
and add other features.
Yeah I expected there may not be many people to review it for mistakes
haha, but luckily it's only ~ 245 lines of Haskell so it shouldn't be
terribly difficult to maintain. I'm definitely planning to maintain this
and will probably start using it at work once we get around to the stateful
side of our architecture. I'll add a note to the README that the API is
still in flux while I iron out a few things and write a test suite for it.
Would definitely be exciting to see Haskell on the list of 3rd party SDKs
though! Glad to see Flink moving in this direction, always have been a huge
fan of the functionality it provides, but wanted a bit more decoupling from
the Flink streaming APIs themselves. This project seems, to me, like
someone has essentially figured out how to do polyglot Akka/OTP at large
scale with internal ACID state (which is my dream architecture).
Thanks,
Tim
On Mon, Sep 14, 2020 at 1:09 PM Igal Shilman <ig...@ververica.com> wrote:
> Hi Tim,
>
> This is amazing! I hope that it wasn't too much trouble for you to figure
> out the bits that needed to be implemented for the RequestReply protocol.
>
> Unfortunately I don't know enough Haskell to provide you any
> meaningful feedback for the SDK itself, not to mention to maintain it at
> this point.
> But what I can suggest is that if you are interested to continue
> maintaining and evolving this, we can add this to a list of 3rd party SDKs
> that are developed
> by the community. And if eventually enough interest would arise and you
> would like to contribute this then we can revisit that!
>
> In any case, this is incredible and I'm looking forward to seeing exciting
> new use cases with Haskell and StateFun :-)
>
> Thanks,
> Igal.
>
>
>
>
> On Mon, Sep 14, 2020 at 3:01 PM Timothy Bess <td...@gmail.com> wrote:
>
>> Hi Gordon and Dawid,
>>
>> Thanks for the responses! I actually ended up just biting the bullet and
>> digging through the Python implementation to figure it out after I wrote
>> that email. Was more worried about accidentally depending on something
>> internal to the Python API that may not be "spec", but it seemed pretty
>> straightforward when I read through it actually. I made a small
>> implementation over the weekend and got the greeter example working. Not
>> sure how familiar anyone is here with Haskell, but here's the code
>> <https://github.com/tdbgamer/flink-statefulfun-hs> if you're interested.
>> The example is pretty simple, just have a function that takes messages and
>> sends them out to a counter ID'd by the user's name.
>>
>> But essentially you interact with the Flink SDK from this one typeclass:
>> [image: image.png]
>>
>>
>> And it essentially just runs through the batches in memory, takes the
>> resulting state, converts it to a "FromFunction" and ships it all back.
>> Haskell has the same sort of green threading concurrency model as Erlang/Go
>> if you're familiar with that, so it was actually very fast when I was
>> testing it.
>>
>> With these sort of SDKs, do people generally get it merged into some
>> Apache/Flink repo and become official? Or does it just stay on my Github?
>>
>> Thanks,
>>
>> Tim
>>
>> On Mon, Sep 14, 2020 at 4:55 AM Tzu-Li (Gordon) Tai <tz...@apache.org>
>> wrote:
>>
>>> Hi!
>>>
>>> Dawid is right, there currently is no developer documentation for the
>>> remote request-reply protocol.
>>> One reason for this is that the protocol isn't considered a fully stable
>>> user-facing interface yet, and thus not yet properly advertised in the
>>> documentation.
>>> However, there are plans to revisit it and announce it as publicly
>>> stable in the near future releases.
>>>
>>> In the meantime, the Python SDK and Aljoscha's Rust SDK are good
>>> reference examples of implementation of the protocol across different
>>> languages.
>>> The request body from Flink and expected response body from functions
>>> are essentially these [1] Protobuf messages, ToFunction (request) and
>>> FromFunction (response).
>>>
>>> If you bump into any issues when implementing, please feel free to let
>>> us know on the mailing lists as well.
>>> A Haskell SDK is definitely interesting to see implemented :)
>>>
>>> Cheers,
>>> Gordon
>>>
>>> [1]
>>> https://github.com/apache/flink-statefun/blob/master/statefun-flink/statefun-flink-core/src/main/protobuf/http-function.proto
>>>
>>> On Mon, Sep 14, 2020 at 3:46 PM Dawid Wysakowicz <dw...@apache.org>
>>> wrote:
>>>
>>>> Hi,
>>>>
>>>> Not sure if there is a "developer" documentation for the protocol. I am
>>>> cc'ing Igal and Gordon who know better than I if there is one.
>>>>
>>>> To give you some hints though. If I am correct the Python API is
>>>> implemented as a so called remote functions [1][2], which communicate
>>>> with Flink via HTTP/gRPC. Besides the bundled Python API you can also
>>>> use a Rust SDK[3] implemented by my colleague Aljoscha as a reference.
>>>>
>>>> BTW, it would be really cool to see an SDK written in Haskell ;)
>>>>
>>>> Best,
>>>>
>>>> Dawid
>>>>
>>>> [1]
>>>>
>>>> https://ci.apache.org/projects/flink/flink-statefun-docs-release-2.1/concepts/distributed_architecture.html#remote-functions
>>>>
>>>> [2]
>>>>
>>>> https://ci.apache.org/projects/flink/flink-statefun-docs-release-2.1/sdk/modules.html#remote-module
>>>>
>>>> [3] https://github.com/aljoscha/statefun-rust
>>>>
>>>> On 12/09/2020 07:26, Timothy Bess wrote:
>>>> > The flink stateful function Python API looks cool, but is there a
>>>> > documented spec for how it communicates with Flink? I'd like to
>>>> > implement an SDK in Haskell if I can.
>>>>
>>>>
Re: Flink Stateful Functions API
Posted by Igal Shilman <ig...@ververica.com>.
Hi Tim,
This is amazing! I hope that it wasn't too much trouble for you to figure
out the bits that needed to be implemented for the RequestReply protocol.
Unfortunately I don't know enough Haskell to provide you any
meaningful feedback for the SDK itself, not to mention to maintain it at
this point.
But what I can suggest is that if you are interested to continue
maintaining and evolving this, we can add this to a list of 3rd party SDKs
that are developed
by the community. And if eventually enough interest would arise and you
would like to contribute this then we can revisit that!
In any case, this is incredible and I'm looking forward to seeing exciting
new use cases with Haskell and StateFun :-)
Thanks,
Igal.
On Mon, Sep 14, 2020 at 3:01 PM Timothy Bess <td...@gmail.com> wrote:
> Hi Gordon and Dawid,
>
> Thanks for the responses! I actually ended up just biting the bullet and
> digging through the Python implementation to figure it out after I wrote
> that email. Was more worried about accidentally depending on something
> internal to the Python API that may not be "spec", but it seemed pretty
> straightforward when I read through it actually. I made a small
> implementation over the weekend and got the greeter example working. Not
> sure how familiar anyone is here with Haskell, but here's the code
> <https://github.com/tdbgamer/flink-statefulfun-hs> if you're interested.
> The example is pretty simple, just have a function that takes messages and
> sends them out to a counter ID'd by the user's name.
>
> But essentially you interact with the Flink SDK from this one typeclass:
> [image: image.png]
>
>
> And it essentially just runs through the batches in memory, takes the
> resulting state, converts it to a "FromFunction" and ships it all back.
> Haskell has the same sort of green threading concurrency model as Erlang/Go
> if you're familiar with that, so it was actually very fast when I was
> testing it.
>
> With these sort of SDKs, do people generally get it merged into some
> Apache/Flink repo and become official? Or does it just stay on my Github?
>
> Thanks,
>
> Tim
>
> On Mon, Sep 14, 2020 at 4:55 AM Tzu-Li (Gordon) Tai <tz...@apache.org>
> wrote:
>
>> Hi!
>>
>> Dawid is right, there currently is no developer documentation for the
>> remote request-reply protocol.
>> One reason for this is that the protocol isn't considered a fully stable
>> user-facing interface yet, and thus not yet properly advertised in the
>> documentation.
>> However, there are plans to revisit it and announce it as publicly stable
>> in the near future releases.
>>
>> In the meantime, the Python SDK and Aljoscha's Rust SDK are good
>> reference examples of implementation of the protocol across different
>> languages.
>> The request body from Flink and expected response body from functions are
>> essentially these [1] Protobuf messages, ToFunction (request) and
>> FromFunction (response).
>>
>> If you bump into any issues when implementing, please feel free to let us
>> know on the mailing lists as well.
>> A Haskell SDK is definitely interesting to see implemented :)
>>
>> Cheers,
>> Gordon
>>
>> [1]
>> https://github.com/apache/flink-statefun/blob/master/statefun-flink/statefun-flink-core/src/main/protobuf/http-function.proto
>>
>> On Mon, Sep 14, 2020 at 3:46 PM Dawid Wysakowicz <dw...@apache.org>
>> wrote:
>>
>>> Hi,
>>>
>>> Not sure if there is a "developer" documentation for the protocol. I am
>>> cc'ing Igal and Gordon who know better than I if there is one.
>>>
>>> To give you some hints though. If I am correct the Python API is
>>> implemented as a so called remote functions [1][2], which communicate
>>> with Flink via HTTP/gRPC. Besides the bundled Python API you can also
>>> use a Rust SDK[3] implemented by my colleague Aljoscha as a reference.
>>>
>>> BTW, it would be really cool to see an SDK written in Haskell ;)
>>>
>>> Best,
>>>
>>> Dawid
>>>
>>> [1]
>>>
>>> https://ci.apache.org/projects/flink/flink-statefun-docs-release-2.1/concepts/distributed_architecture.html#remote-functions
>>>
>>> [2]
>>>
>>> https://ci.apache.org/projects/flink/flink-statefun-docs-release-2.1/sdk/modules.html#remote-module
>>>
>>> [3] https://github.com/aljoscha/statefun-rust
>>>
>>> On 12/09/2020 07:26, Timothy Bess wrote:
>>> > The flink stateful function Python API looks cool, but is there a
>>> > documented spec for how it communicates with Flink? I'd like to
>>> > implement an SDK in Haskell if I can.
>>>
>>>
Re: Flink Stateful Functions API
Posted by Timothy Bess <td...@gmail.com>.
Hi Gordon and Dawid,
Thanks for the responses! I actually ended up just biting the bullet and
digging through the Python implementation to figure it out after I wrote
that email. Was more worried about accidentally depending on something
internal to the Python API that may not be "spec", but it seemed pretty
straightforward when I read through it actually. I made a small
implementation over the weekend and got the greeter example working. Not
sure how familiar anyone is here with Haskell, but here's the code
<https://github.com/tdbgamer/flink-statefulfun-hs> if you're interested.
The example is pretty simple, just have a function that takes messages and
sends them out to a counter ID'd by the user's name.
But essentially you interact with the Flink SDK from this one typeclass:
[image: image.png]
And it essentially just runs through the batches in memory, takes the
resulting state, converts it to a "FromFunction" and ships it all back.
Haskell has the same sort of green threading concurrency model as Erlang/Go
if you're familiar with that, so it was actually very fast when I was
testing it.
With these sort of SDKs, do people generally get it merged into some
Apache/Flink repo and become official? Or does it just stay on my Github?
Thanks,
Tim
On Mon, Sep 14, 2020 at 4:55 AM Tzu-Li (Gordon) Tai <tz...@apache.org>
wrote:
> Hi!
>
> Dawid is right, there currently is no developer documentation for the
> remote request-reply protocol.
> One reason for this is that the protocol isn't considered a fully stable
> user-facing interface yet, and thus not yet properly advertised in the
> documentation.
> However, there are plans to revisit it and announce it as publicly stable
> in the near future releases.
>
> In the meantime, the Python SDK and Aljoscha's Rust SDK are good
> reference examples of implementation of the protocol across different
> languages.
> The request body from Flink and expected response body from functions are
> essentially these [1] Protobuf messages, ToFunction (request) and
> FromFunction (response).
>
> If you bump into any issues when implementing, please feel free to let us
> know on the mailing lists as well.
> A Haskell SDK is definitely interesting to see implemented :)
>
> Cheers,
> Gordon
>
> [1]
> https://github.com/apache/flink-statefun/blob/master/statefun-flink/statefun-flink-core/src/main/protobuf/http-function.proto
>
> On Mon, Sep 14, 2020 at 3:46 PM Dawid Wysakowicz <dw...@apache.org>
> wrote:
>
>> Hi,
>>
>> Not sure if there is a "developer" documentation for the protocol. I am
>> cc'ing Igal and Gordon who know better than I if there is one.
>>
>> To give you some hints though. If I am correct the Python API is
>> implemented as a so called remote functions [1][2], which communicate
>> with Flink via HTTP/gRPC. Besides the bundled Python API you can also
>> use a Rust SDK[3] implemented by my colleague Aljoscha as a reference.
>>
>> BTW, it would be really cool to see an SDK written in Haskell ;)
>>
>> Best,
>>
>> Dawid
>>
>> [1]
>>
>> https://ci.apache.org/projects/flink/flink-statefun-docs-release-2.1/concepts/distributed_architecture.html#remote-functions
>>
>> [2]
>>
>> https://ci.apache.org/projects/flink/flink-statefun-docs-release-2.1/sdk/modules.html#remote-module
>>
>> [3] https://github.com/aljoscha/statefun-rust
>>
>> On 12/09/2020 07:26, Timothy Bess wrote:
>> > The flink stateful function Python API looks cool, but is there a
>> > documented spec for how it communicates with Flink? I'd like to
>> > implement an SDK in Haskell if I can.
>>
>>
Re: Flink Stateful Functions API
Posted by "Tzu-Li (Gordon) Tai" <tz...@apache.org>.
Hi!
Dawid is right, there currently is no developer documentation for the
remote request-reply protocol.
One reason for this is that the protocol isn't considered a fully stable
user-facing interface yet, and thus not yet properly advertised in the
documentation.
However, there are plans to revisit it and announce it as publicly stable
in the near future releases.
In the meantime, the Python SDK and Aljoscha's Rust SDK are good
reference examples of implementation of the protocol across different
languages.
The request body from Flink and expected response body from functions are
essentially these [1] Protobuf messages, ToFunction (request) and
FromFunction (response).
If you bump into any issues when implementing, please feel free to let us
know on the mailing lists as well.
A Haskell SDK is definitely interesting to see implemented :)
Cheers,
Gordon
[1]
https://github.com/apache/flink-statefun/blob/master/statefun-flink/statefun-flink-core/src/main/protobuf/http-function.proto
On Mon, Sep 14, 2020 at 3:46 PM Dawid Wysakowicz <dw...@apache.org>
wrote:
> Hi,
>
> Not sure if there is a "developer" documentation for the protocol. I am
> cc'ing Igal and Gordon who know better than I if there is one.
>
> To give you some hints though. If I am correct the Python API is
> implemented as a so called remote functions [1][2], which communicate
> with Flink via HTTP/gRPC. Besides the bundled Python API you can also
> use a Rust SDK[3] implemented by my colleague Aljoscha as a reference.
>
> BTW, it would be really cool to see an SDK written in Haskell ;)
>
> Best,
>
> Dawid
>
> [1]
>
> https://ci.apache.org/projects/flink/flink-statefun-docs-release-2.1/concepts/distributed_architecture.html#remote-functions
>
> [2]
>
> https://ci.apache.org/projects/flink/flink-statefun-docs-release-2.1/sdk/modules.html#remote-module
>
> [3] https://github.com/aljoscha/statefun-rust
>
> On 12/09/2020 07:26, Timothy Bess wrote:
> > The flink stateful function Python API looks cool, but is there a
> > documented spec for how it communicates with Flink? I'd like to
> > implement an SDK in Haskell if I can.
>
>
Re: Flink Stateful Functions API
Posted by Dawid Wysakowicz <dw...@apache.org>.
Hi,
Not sure if there is a "developer" documentation for the protocol. I am
cc'ing Igal and Gordon who know better than I if there is one.
To give you some hints though. If I am correct the Python API is
implemented as a so called remote functions [1][2], which communicate
with Flink via HTTP/gRPC. Besides the bundled Python API you can also
use a Rust SDK[3] implemented by my colleague Aljoscha as a reference.
BTW, it would be really cool to see an SDK written in Haskell ;)
Best,
Dawid
[1]
https://ci.apache.org/projects/flink/flink-statefun-docs-release-2.1/concepts/distributed_architecture.html#remote-functions
[2]
https://ci.apache.org/projects/flink/flink-statefun-docs-release-2.1/sdk/modules.html#remote-module
[3] https://github.com/aljoscha/statefun-rust
On 12/09/2020 07:26, Timothy Bess wrote:
> The flink stateful function Python API looks cool, but is there a
> documented spec for how it communicates with Flink? I'd like to
> implement an SDK in Haskell if I can.