Re: how to connect to the flink-state store and use it as cache to serve APIs.

[laxmi narayan <ni...@gmail.com>]

Hi Folks,

I just wanted to double check, if there is any way to expose rest APIs
using Flink sql tables ?




Thank you.


On Thu, Jun 30, 2022 at 12:15 PM Yuan Mei <yu...@gmail.com> wrote:

> That's definitely something we want to achieve in the future term, and
> your input is very valuable.
>
> One problem with the current queryable state setup is that the service is
> bounded to the life cycle of Flink Job, which limits the usage of the state
> store/service.
>
> Thanks for your insights.
>
> Best
> Yuan
>
> On Wed, Jun 29, 2022 at 3:41 PM laxmi narayan <ni...@gmail.com>
> wrote:
>
>>
>> Hi Hangxiang,
>>
>> I was thinking , since we already store entire state in the checkpoint
>> dir so why can't we expose it as a service through the Flink queryable
>> state, in this way I can easily avoid introducing a cache and serve
>> realtime APIs via this state itself and I can go to the database for the
>> historical data.
>>
>>
>>
>> Thank you.
>>
>>
>> On Wed, Jun 29, 2022 at 11:17 AM Hangxiang Yu <ma...@gmail.com>
>> wrote:
>>
>>> Hi, laxmi.
>>> There are two ways that users can access the state store currently:
>>> 1. Queryable state [1] which you could access states in runtime.
>>> 2. State Processor API [2] which you could access states (snapshot)
>>> offline.
>>>
>>> But we have marked the Queryable state as "Reaching End-of-Life".
>>> We are also trying to find a graceful and effective way for users to
>>> debug and troubleshoot.
>>> So could you share your case about what you want to use it for ?
>>> Your feedback is important for us to design it in the long term. Thanks!
>>>
>>> [1]
>>> https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/fault-tolerance/queryable_state/
>>> [2]
>>> https://nightlies.apache.org/flink/flink-docs-master/docs/libs/state_processor_api/
>>> [3] https://flink.apache.org/roadmap.html
>>>
>>> Best,
>>> Hangxiang.
>>>
>>> On Tue, Jun 28, 2022 at 8:26 PM laxmi narayan <ni...@gmail.com>
>>> wrote:
>>>
>>>> Hi Team,
>>>> I am not sure if this is the right use case for the state-store but I
>>>> wanted to serve the APIs using queryable-state, what are the different ways
>>>> to achieve this ?
>>>>
>>>> I have come across a version where we can use Job_Id to connect to the
>>>> state, but is there any other way to expose a specific rest-endpoint etc ?
>>>>
>>>> Any sample example/github link would be nice.
>>>>
>>>>
>>>>
>>>> Thank you.
>>>>
>>>

RE: how to connect to the flink-state store and use it as cache to serve APIs.

[Schwalbe Matthias <Ma...@viseca.ch>]

Hi Laxmi,

Did you consider Apache Flink Table Store [1] which was introduced short time ago.
Yours sounds like a case for early integration …

Sincere greetings

Thias

[1] https://nightlies.apache.org/flink/flink-table-store-docs-release-0.1/

From: laxmi narayan <ni...@gmail.com>
Sent: Wednesday, July 6, 2022 6:29 AM
To: Yuan Mei <yu...@gmail.com>
Cc: Hangxiang Yu <ma...@gmail.com>; user <us...@flink.apache.org>
Subject: Re: how to connect to the flink-state store and use it as cache to serve APIs.

⚠EXTERNAL MESSAGE – CAUTION: Think Before You Click ⚠



Hi Folks,

I just wanted to double check, if there is any way to expose rest APIs using Flink sql tables ?




Thank you.


On Thu, Jun 30, 2022 at 12:15 PM Yuan Mei <yu...@gmail.com>> wrote:
That's definitely something we want to achieve in the future term, and your input is very valuable.

One problem with the current queryable state setup is that the service is bounded to the life cycle of Flink Job, which limits the usage of the state store/service.

Thanks for your insights.

Best
Yuan

On Wed, Jun 29, 2022 at 3:41 PM laxmi narayan <ni...@gmail.com>> wrote:

Hi Hangxiang,

I was thinking , since we already store entire state in the checkpoint dir so why can't we expose it as a service through the Flink queryable state, in this way I can easily avoid introducing a cache and serve realtime APIs via this state itself and I can go to the database for the historical data.



Thank you.


On Wed, Jun 29, 2022 at 11:17 AM Hangxiang Yu <ma...@gmail.com>> wrote:
Hi, laxmi.
There are two ways that users can access the state store currently:
1. Queryable state [1] which you could access states in runtime.
2. State Processor API [2] which you could access states (snapshot) offline.

But we have marked the Queryable state as "Reaching End-of-Life".
We are also trying to find a graceful and effective way for users to debug and troubleshoot.
So could you share your case about what you want to use it for ?
Your feedback is important for us to design it in the long term. Thanks!

[1] https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/fault-tolerance/queryable_state/
[2] https://nightlies.apache.org/flink/flink-docs-master/docs/libs/state_processor_api/
[3] https://flink.apache.org/roadmap.html

Best,
Hangxiang.

On Tue, Jun 28, 2022 at 8:26 PM laxmi narayan <ni...@gmail.com>> wrote:
Hi Team,
I am not sure if this is the right use case for the state-store but I wanted to serve the APIs using queryable-state, what are the different ways to achieve this ?

I have come across a version where we can use Job_Id to connect to the state, but is there any other way to expose a specific rest-endpoint etc ?
Any sample example/github link would be nice.



Thank you.
Diese Nachricht ist ausschliesslich für den Adressaten bestimmt und beinhaltet unter Umständen vertrauliche Mitteilungen. Da die Vertraulichkeit von e-Mail-Nachrichten nicht gewährleistet werden kann, übernehmen wir keine Haftung für die Gewährung der Vertraulichkeit und Unversehrtheit dieser Mitteilung. Bei irrtümlicher Zustellung bitten wir Sie um Benachrichtigung per e-Mail und um Löschung dieser Nachricht sowie eventueller Anhänge. Jegliche unberechtigte Verwendung oder Verbreitung dieser Informationen ist streng verboten.

This message is intended only for the named recipient and may contain confidential or privileged information. As the confidentiality of email communication cannot be guaranteed, we do not accept any responsibility for the confidentiality and the intactness of this message. If you have received it in error, please advise the sender by return e-mail and delete this message and any attachments. Any unauthorised use or dissemination of this information is strictly prohibited.