You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Jessy Ping <te...@gmail.com> on 2022/01/24 15:46:33 UTC
Regarding Queryable state in Flink
Hi Team,
We are currently running our streaming application based Flink(Datastream
API ) on a non-prod cluster.And planning to move it to production cluster
soon.. We are keeping cerating keyed state backed by rocksdb in the flink
application. We need a mechanism to query these keyed state values for
debugging and troubleshooting. Is it a good idea to use Queryable state for
a single link-job running in application-mode on kubernetes for an average
load of 10k events/second ?.
Or is it a better idea to keep these state values in an external k,v store
?.
So in short --> Is the queryable state stable enough to use in production
systems ?
Thanks
Jessy
Re: Regarding Queryable state in Flink
Posted by Martijn Visser <ma...@ververica.com>.
Hi Jessy,
Queryable State is considered approaching end of life [1] per the Flink
Roadmap.
There are currently no development activities planned for it.
Best regards,
Martijn
[1]
https://flink.apache.org/roadmap.html
Op di 25 jan. 2022 om 18:00 schreef Jessy Ping <te...@gmail.com>
> Hi Matthias,
>
> I want to query the current state of the application at real-time. Hence,
> state processor API won't fit here. I have the following questions,
>
> * Is the queryable state stable enough to use in production systems ?.
>
> Are there any improvements or development activities planned or going on
> with queryable state ? Is Flink community still supporting queryable state
> and suggesting its usage? .
>
>
>
> Thanks Jessy
>
>
> On Tue, Jan 25, 2022, 10:19 PM Schwalbe Matthias <
> Matthias.Schwalbe@viseca.ch> wrote:
>
>> Hi Jessy,
>>
>>
>>
>> Have you considered using the state processor api [1] for offline
>> analysis of checkpoints and savepoints?
>>
>>
>>
>> [1]
>> https://nightlies.apache.org/flink/flink-docs-release-1.13/docs/libs/state_processor_api/
>>
>>
>>
>> Sincere greetings
>>
>>
>>
>> Thias
>>
>>
>>
>>
>>
>> *From:* Jessy Ping <te...@gmail.com>
>> *Sent:* Montag, 24. Januar 2022 16:47
>> *To:* user <us...@flink.apache.org>
>> *Subject:* Regarding Queryable state in Flink
>>
>>
>>
>> Hi Team,
>>
>>
>>
>> We are currently running our streaming application based Flink(Datastream
>> API ) on a non-prod cluster.And planning to move it to production cluster
>> soon.. We are keeping cerating keyed state backed by rocksdb in the flink
>> application. We need a mechanism to query these keyed state values for
>> debugging and troubleshooting. Is it a good idea to use Queryable state for
>> a single link-job running in application-mode on kubernetes for an average
>> load of 10k events/second ?.
>>
>> Or is it a better idea to keep these state values in an external k,v
>> store ?.
>>
>>
>>
>> So in short --> Is the queryable state stable enough to use in production
>> systems ?
>>
>>
>>
>>
>>
>> Thanks
>>
>> Jessy
>> Diese Nachricht ist ausschliesslich für den Adressaten bestimmt und
>> beinhaltet unter Umständen vertrauliche Mitteilungen. Da die
>> Vertraulichkeit von e-Mail-Nachrichten nicht gewährleistet werden kann,
>> übernehmen wir keine Haftung für die Gewährung der Vertraulichkeit und
>> Unversehrtheit dieser Mitteilung. Bei irrtümlicher Zustellung bitten wir
>> Sie um Benachrichtigung per e-Mail und um Löschung dieser Nachricht sowie
>> eventueller Anhänge. Jegliche unberechtigte Verwendung oder Verbreitung
>> dieser Informationen ist streng verboten.
>>
>> This message is intended only for the named recipient and may contain
>> confidential or privileged information. As the confidentiality of email
>> communication cannot be guaranteed, we do not accept any responsibility for
>> the confidentiality and the intactness of this message. If you have
>> received it in error, please advise the sender by return e-mail and delete
>> this message and any attachments. Any unauthorised use or dissemination of
>> this information is strictly prohibited.
>>
> --
Martijn Visser | Product Manager
martijn@ververica.com
<https://www.ververica.com/>
Follow us @VervericaData
--
Join Flink Forward <https://flink-forward.org/> - The Apache Flink
Conference
Stream Processing | Event Driven | Real Time
Re: Regarding Queryable state in Flink
Posted by Jessy Ping <te...@gmail.com>.
Hi Matthias,
I want to query the current state of the application at real-time. Hence,
state processor API won't fit here. I have the following questions,
* Is the queryable state stable enough to use in production systems ?.
Are there any improvements or development activities planned or going on
with queryable state ? Is Flink community still supporting queryable state
and suggesting its usage? .
Thanks Jessy
On Tue, Jan 25, 2022, 10:19 PM Schwalbe Matthias <
Matthias.Schwalbe@viseca.ch> wrote:
> Hi Jessy,
>
>
>
> Have you considered using the state processor api [1] for offline analysis
> of checkpoints and savepoints?
>
>
>
> [1]
> https://nightlies.apache.org/flink/flink-docs-release-1.13/docs/libs/state_processor_api/
>
>
>
> Sincere greetings
>
>
>
> Thias
>
>
>
>
>
> *From:* Jessy Ping <te...@gmail.com>
> *Sent:* Montag, 24. Januar 2022 16:47
> *To:* user <us...@flink.apache.org>
> *Subject:* Regarding Queryable state in Flink
>
>
>
> Hi Team,
>
>
>
> We are currently running our streaming application based Flink(Datastream
> API ) on a non-prod cluster.And planning to move it to production cluster
> soon.. We are keeping cerating keyed state backed by rocksdb in the flink
> application. We need a mechanism to query these keyed state values for
> debugging and troubleshooting. Is it a good idea to use Queryable state for
> a single link-job running in application-mode on kubernetes for an average
> load of 10k events/second ?.
>
> Or is it a better idea to keep these state values in an external k,v store
> ?.
>
>
>
> So in short --> Is the queryable state stable enough to use in production
> systems ?
>
>
>
>
>
> Thanks
>
> Jessy
> Diese Nachricht ist ausschliesslich für den Adressaten bestimmt und
> beinhaltet unter Umständen vertrauliche Mitteilungen. Da die
> Vertraulichkeit von e-Mail-Nachrichten nicht gewährleistet werden kann,
> übernehmen wir keine Haftung für die Gewährung der Vertraulichkeit und
> Unversehrtheit dieser Mitteilung. Bei irrtümlicher Zustellung bitten wir
> Sie um Benachrichtigung per e-Mail und um Löschung dieser Nachricht sowie
> eventueller Anhänge. Jegliche unberechtigte Verwendung oder Verbreitung
> dieser Informationen ist streng verboten.
>
> This message is intended only for the named recipient and may contain
> confidential or privileged information. As the confidentiality of email
> communication cannot be guaranteed, we do not accept any responsibility for
> the confidentiality and the intactness of this message. If you have
> received it in error, please advise the sender by return e-mail and delete
> this message and any attachments. Any unauthorised use or dissemination of
> this information is strictly prohibited.
>
RE: Regarding Queryable state in Flink
Posted by Schwalbe Matthias <Ma...@viseca.ch>.
Hi Jessy,
Have you considered using the state processor api [1] for offline analysis of checkpoints and savepoints?
[1] https://nightlies.apache.org/flink/flink-docs-release-1.13/docs/libs/state_processor_api/
Sincere greetings
Thias
From: Jessy Ping <te...@gmail.com>
Sent: Montag, 24. Januar 2022 16:47
To: user <us...@flink.apache.org>
Subject: Regarding Queryable state in Flink
Hi Team,
We are currently running our streaming application based Flink(Datastream API ) on a non-prod cluster.And planning to move it to production cluster soon.. We are keeping cerating keyed state backed by rocksdb in the flink application. We need a mechanism to query these keyed state values for debugging and troubleshooting. Is it a good idea to use Queryable state for a single link-job running in application-mode on kubernetes for an average load of 10k events/second ?.
Or is it a better idea to keep these state values in an external k,v store ?.
So in short --> Is the queryable state stable enough to use in production systems ?
Thanks
Jessy
Diese Nachricht ist ausschliesslich für den Adressaten bestimmt und beinhaltet unter Umständen vertrauliche Mitteilungen. Da die Vertraulichkeit von e-Mail-Nachrichten nicht gewährleistet werden kann, übernehmen wir keine Haftung für die Gewährung der Vertraulichkeit und Unversehrtheit dieser Mitteilung. Bei irrtümlicher Zustellung bitten wir Sie um Benachrichtigung per e-Mail und um Löschung dieser Nachricht sowie eventueller Anhänge. Jegliche unberechtigte Verwendung oder Verbreitung dieser Informationen ist streng verboten.
This message is intended only for the named recipient and may contain confidential or privileged information. As the confidentiality of email communication cannot be guaranteed, we do not accept any responsibility for the confidentiality and the intactness of this message. If you have received it in error, please advise the sender by return e-mail and delete this message and any attachments. Any unauthorised use or dissemination of this information is strictly prohibited.