You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Vishal Santoshi <vi...@gmail.com> on 2018/03/16 21:10:59 UTC
Queryable State
We are making few decisions on use cases where Queryable state is a
natural fit
https://ci.apache.org/projects/flink/flink-docs-release-1.4/dev/stream/state/queryable_state.html
Is Queryable state production ready ? We will go to 1.5 flnk if that helps
to make the case for the usage.
Re: Queryable State
Posted by Vishal Santoshi <vi...@gmail.com>.
Thank you for the clarification.
On Wed, Mar 21, 2018, 4:28 AM Kostas Kloudas <k....@data-artisans.com>
wrote:
> Hi Vishal,
>
> As Fabian said, queryable state is just a feature that exposes the state
> kept within Flink, and it is not made to
> replace functionality that would otherwise be made by a sink. In the
> future the functionality will definitely evolve
> but for there are no discussions currently, for keeping the state of a job
> even after the job is done.
>
> For being able to do so, with exactly once semantics and all the
> guarantees provided by Flink, I would recommend
> to use an external sink.
>
> Cheers,
> Kostas
>
> On Mar 19, 2018, at 6:18 PM, Vishal Santoshi <vi...@gmail.com>
> wrote:
>
> Thank you. These do look like show stoppers for us. But again thank you.
>
> On Mon, Mar 19, 2018 at 10:31 AM, Fabian Hueske <fh...@gmail.com> wrote:
>
>> AFAIK, there have been discussions to replicate state among TMs to speed
>> up recovery (and improve availability).
>> However, I'm not aware of plans to implement that.
>>
>> I don't think serving state while a job is down has been considered yet.
>>
>> 2018-03-19 15:17 GMT+01:00 Vishal Santoshi <vi...@gmail.com>:
>>
>>> Are there plans to address all or few of the above apart from the "JM
>>> LB not possible" which seems understandable ?
>>>
>>> On Mon, Mar 19, 2018 at 9:58 AM, Fabian Hueske <fh...@gmail.com>
>>> wrote:
>>>
>>>> Queryable state is "just" an additional feature on regular keyed state.
>>>> i.e., the only difference is that you can read the state from an outside
>>>> application.
>>>> Besides that it behaves exactly like regular application state
>>>>
>>>> Queryable state is (at the moment) designed to be accessible if a job
>>>> runs.
>>>> If the job fails (and recovers) or is manually taken down for
>>>> maintenance, the state cannot be queried.
>>>> It's not possible to put a load balancer in front of a JobManager. Only
>>>> one JM is the active master that maintains a running job.
>>>> State is also not replicated.
>>>>
>>>> Best, Fabian
>>>>
>>>>
>>>> 2018-03-19 14:24 GMT+01:00 Vishal Santoshi <vi...@gmail.com>:
>>>>
>>>>> Those are understandable. I am more interested in a few things ( and
>>>>> may be more that could be added )
>>>>>
>>>>> * As far as I can understand JM is the SPOF. Does HA become a
>>>>> necessity ?
>>>>> * If there are 2 or more JM could we theoretically have a LB fronting
>>>>> them ? Thus it is a peer to peer access ( Cassandra ) or a master slave
>>>>> setup for JM HA specifically for Queryable access ( For flink jobs it is
>>>>> master slave )
>>>>> * Do we replicate state to other TMs for read optimization (
>>>>> specifically to avoid Hot Node issues ) ?
>>>>> * If the job goes down it seems the state is not accessible. What
>>>>> plans to we have to "separate concerns" for Queryable state.
>>>>>
>>>>> We consider Queryable State significant a feature Flink provides and
>>>>> would do the necessary leg work if there are certain gaps in it being
>>>>> trully considered a Highly Available Key Value store.
>>>>>
>>>>> Regards.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Mon, Mar 19, 2018 at 5:53 AM, Fabian Hueske <fh...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Hi Vishal,
>>>>>>
>>>>>> In general, Queryable State should be ready to use.
>>>>>> There are a few things to consider though:
>>>>>>
>>>>>> - State queries are not synchronized with the application code, i.e.,
>>>>>> they can happen at the same time. Therefore, the Flink application should
>>>>>> not modify objects that have been put into or read from the state if you
>>>>>> are not using the RocksDBStatebackend (which creates copies by
>>>>>> deserialization).
>>>>>> - State will be rolled back after a failure. Hence, you can read
>>>>>> writes that are not "committed by a checkpoint".
>>>>>>
>>>>>> @Kostas, did I forget something?
>>>>>>
>>>>>> Best, Fabian
>>>>>>
>>>>>>
>>>>>>
>>>>>> 2018-03-18 16:50 GMT+01:00 Vishal Santoshi <vishal.santoshi@gmail.com
>>>>>> >:
>>>>>>
>>>>>>> To be more precise, is anything thing similar to
>>>>>>> https://engineering.linkedin.com/blog/2018/03/air-traffic-controller--member-first-notifications-at-linkedin
>>>>>>> . done in Samza, can be replicated with production level guarantees with
>>>>>>> Flink Queryable state ( as it stands currently version 1.5 ) ?
>>>>>>>
>>>>>>> On Fri, Mar 16, 2018 at 5:10 PM, Vishal Santoshi <
>>>>>>> vishal.santoshi@gmail.com> wrote:
>>>>>>>
>>>>>>>> We are making few decisions on use cases where Queryable state is
>>>>>>>> a natural fit
>>>>>>>> https://ci.apache.org/projects/flink/flink-docs-release-1.4/dev/stream/state/queryable_state.html
>>>>>>>>
>>>>>>>> Is Queryable state production ready ? We will go to 1.5 flnk if
>>>>>>>> that helps to make the case for the usage.
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
>
Re: Queryable State
Posted by Kostas Kloudas <k....@data-artisans.com>.
Hi Vishal,
As Fabian said, queryable state is just a feature that exposes the state kept within Flink, and it is not made to
replace functionality that would otherwise be made by a sink. In the future the functionality will definitely evolve
but for there are no discussions currently, for keeping the state of a job even after the job is done.
For being able to do so, with exactly once semantics and all the guarantees provided by Flink, I would recommend
to use an external sink.
Cheers,
Kostas
> On Mar 19, 2018, at 6:18 PM, Vishal Santoshi <vi...@gmail.com> wrote:
>
> Thank you. These do look like show stoppers for us. But again thank you.
>
> On Mon, Mar 19, 2018 at 10:31 AM, Fabian Hueske <fhueske@gmail.com <ma...@gmail.com>> wrote:
> AFAIK, there have been discussions to replicate state among TMs to speed up recovery (and improve availability).
> However, I'm not aware of plans to implement that.
>
> I don't think serving state while a job is down has been considered yet.
>
> 2018-03-19 15:17 GMT+01:00 Vishal Santoshi <vishal.santoshi@gmail.com <ma...@gmail.com>>:
> Are there plans to address all or few of the above apart from the "JM LB not possible" which seems understandable ?
>
> On Mon, Mar 19, 2018 at 9:58 AM, Fabian Hueske <fhueske@gmail.com <ma...@gmail.com>> wrote:
> Queryable state is "just" an additional feature on regular keyed state. i.e., the only difference is that you can read the state from an outside application.
> Besides that it behaves exactly like regular application state
>
> Queryable state is (at the moment) designed to be accessible if a job runs.
> If the job fails (and recovers) or is manually taken down for maintenance, the state cannot be queried.
> It's not possible to put a load balancer in front of a JobManager. Only one JM is the active master that maintains a running job.
> State is also not replicated.
>
> Best, Fabian
>
>
> 2018-03-19 14:24 GMT+01:00 Vishal Santoshi <vishal.santoshi@gmail.com <ma...@gmail.com>>:
> Those are understandable. I am more interested in a few things ( and may be more that could be added )
>
> * As far as I can understand JM is the SPOF. Does HA become a necessity ?
> * If there are 2 or more JM could we theoretically have a LB fronting them ? Thus it is a peer to peer access ( Cassandra ) or a master slave setup for JM HA specifically for Queryable access ( For flink jobs it is master slave )
> * Do we replicate state to other TMs for read optimization ( specifically to avoid Hot Node issues ) ?
> * If the job goes down it seems the state is not accessible. What plans to we have to "separate concerns" for Queryable state.
>
> We consider Queryable State significant a feature Flink provides and would do the necessary leg work if there are certain gaps in it being trully considered a Highly Available Key Value store.
>
> Regards.
>
>
>
>
>
> On Mon, Mar 19, 2018 at 5:53 AM, Fabian Hueske <fhueske@gmail.com <ma...@gmail.com>> wrote:
> Hi Vishal,
>
> In general, Queryable State should be ready to use.
> There are a few things to consider though:
>
> - State queries are not synchronized with the application code, i.e., they can happen at the same time. Therefore, the Flink application should not modify objects that have been put into or read from the state if you are not using the RocksDBStatebackend (which creates copies by deserialization).
> - State will be rolled back after a failure. Hence, you can read writes that are not "committed by a checkpoint".
>
> @Kostas, did I forget something?
>
> Best, Fabian
>
>
>
> 2018-03-18 16:50 GMT+01:00 Vishal Santoshi <vishal.santoshi@gmail.com <ma...@gmail.com>>:
> To be more precise, is anything thing similar to https://engineering.linkedin.com/blog/2018/03/air-traffic-controller--member-first-notifications-at-linkedin <https://engineering.linkedin.com/blog/2018/03/air-traffic-controller--member-first-notifications-at-linkedin> . done in Samza, can be replicated with production level guarantees with Flink Queryable state ( as it stands currently version 1.5 ) ?
>
> On Fri, Mar 16, 2018 at 5:10 PM, Vishal Santoshi <vishal.santoshi@gmail.com <ma...@gmail.com>> wrote:
> We are making few decisions on use cases where Queryable state is a natural fit https://ci.apache.org/projects/flink/flink-docs-release-1.4/dev/stream/state/queryable_state.html <https://ci.apache.org/projects/flink/flink-docs-release-1.4/dev/stream/state/queryable_state.html>
>
> Is Queryable state production ready ? We will go to 1.5 flnk if that helps to make the case for the usage.
>
>
>
>
>
>
>
Re: Queryable State
Posted by Vishal Santoshi <vi...@gmail.com>.
Thank you. These do look like show stoppers for us. But again thank you.
On Mon, Mar 19, 2018 at 10:31 AM, Fabian Hueske <fh...@gmail.com> wrote:
> AFAIK, there have been discussions to replicate state among TMs to speed
> up recovery (and improve availability).
> However, I'm not aware of plans to implement that.
>
> I don't think serving state while a job is down has been considered yet.
>
> 2018-03-19 15:17 GMT+01:00 Vishal Santoshi <vi...@gmail.com>:
>
>> Are there plans to address all or few of the above apart from the "JM LB
>> not possible" which seems understandable ?
>>
>> On Mon, Mar 19, 2018 at 9:58 AM, Fabian Hueske <fh...@gmail.com> wrote:
>>
>>> Queryable state is "just" an additional feature on regular keyed state.
>>> i.e., the only difference is that you can read the state from an outside
>>> application.
>>> Besides that it behaves exactly like regular application state
>>>
>>> Queryable state is (at the moment) designed to be accessible if a job
>>> runs.
>>> If the job fails (and recovers) or is manually taken down for
>>> maintenance, the state cannot be queried.
>>> It's not possible to put a load balancer in front of a JobManager. Only
>>> one JM is the active master that maintains a running job.
>>> State is also not replicated.
>>>
>>> Best, Fabian
>>>
>>>
>>> 2018-03-19 14:24 GMT+01:00 Vishal Santoshi <vi...@gmail.com>:
>>>
>>>> Those are understandable. I am more interested in a few things ( and
>>>> may be more that could be added )
>>>>
>>>> * As far as I can understand JM is the SPOF. Does HA become a necessity
>>>> ?
>>>> * If there are 2 or more JM could we theoretically have a LB fronting
>>>> them ? Thus it is a peer to peer access ( Cassandra ) or a master slave
>>>> setup for JM HA specifically for Queryable access ( For flink jobs it is
>>>> master slave )
>>>> * Do we replicate state to other TMs for read optimization (
>>>> specifically to avoid Hot Node issues ) ?
>>>> * If the job goes down it seems the state is not accessible. What plans
>>>> to we have to "separate concerns" for Queryable state.
>>>>
>>>> We consider Queryable State significant a feature Flink provides and
>>>> would do the necessary leg work if there are certain gaps in it being
>>>> trully considered a Highly Available Key Value store.
>>>>
>>>> Regards.
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> On Mon, Mar 19, 2018 at 5:53 AM, Fabian Hueske <fh...@gmail.com>
>>>> wrote:
>>>>
>>>>> Hi Vishal,
>>>>>
>>>>> In general, Queryable State should be ready to use.
>>>>> There are a few things to consider though:
>>>>>
>>>>> - State queries are not synchronized with the application code, i.e.,
>>>>> they can happen at the same time. Therefore, the Flink application should
>>>>> not modify objects that have been put into or read from the state if you
>>>>> are not using the RocksDBStatebackend (which creates copies by
>>>>> deserialization).
>>>>> - State will be rolled back after a failure. Hence, you can read
>>>>> writes that are not "committed by a checkpoint".
>>>>>
>>>>> @Kostas, did I forget something?
>>>>>
>>>>> Best, Fabian
>>>>>
>>>>>
>>>>>
>>>>> 2018-03-18 16:50 GMT+01:00 Vishal Santoshi <vi...@gmail.com>
>>>>> :
>>>>>
>>>>>> To be more precise, is anything thing similar to
>>>>>> https://engineering.linkedin.com/blog/2018/03/air-traffic
>>>>>> -controller--member-first-notifications-at-linkedin . done in Samza,
>>>>>> can be replicated with production level guarantees with Flink Queryable
>>>>>> state ( as it stands currently version 1.5 ) ?
>>>>>>
>>>>>> On Fri, Mar 16, 2018 at 5:10 PM, Vishal Santoshi <
>>>>>> vishal.santoshi@gmail.com> wrote:
>>>>>>
>>>>>>> We are making few decisions on use cases where Queryable state is a
>>>>>>> natural fit https://ci.apache.org/projects
>>>>>>> /flink/flink-docs-release-1.4/dev/stream/state/queryable_state.html
>>>>>>>
>>>>>>> Is Queryable state production ready ? We will go to 1.5 flnk if that
>>>>>>> helps to make the case for the usage.
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
Re: Queryable State
Posted by Fabian Hueske <fh...@gmail.com>.
AFAIK, there have been discussions to replicate state among TMs to speed up
recovery (and improve availability).
However, I'm not aware of plans to implement that.
I don't think serving state while a job is down has been considered yet.
2018-03-19 15:17 GMT+01:00 Vishal Santoshi <vi...@gmail.com>:
> Are there plans to address all or few of the above apart from the "JM LB
> not possible" which seems understandable ?
>
> On Mon, Mar 19, 2018 at 9:58 AM, Fabian Hueske <fh...@gmail.com> wrote:
>
>> Queryable state is "just" an additional feature on regular keyed state.
>> i.e., the only difference is that you can read the state from an outside
>> application.
>> Besides that it behaves exactly like regular application state
>>
>> Queryable state is (at the moment) designed to be accessible if a job
>> runs.
>> If the job fails (and recovers) or is manually taken down for
>> maintenance, the state cannot be queried.
>> It's not possible to put a load balancer in front of a JobManager. Only
>> one JM is the active master that maintains a running job.
>> State is also not replicated.
>>
>> Best, Fabian
>>
>>
>> 2018-03-19 14:24 GMT+01:00 Vishal Santoshi <vi...@gmail.com>:
>>
>>> Those are understandable. I am more interested in a few things ( and may
>>> be more that could be added )
>>>
>>> * As far as I can understand JM is the SPOF. Does HA become a necessity ?
>>> * If there are 2 or more JM could we theoretically have a LB fronting
>>> them ? Thus it is a peer to peer access ( Cassandra ) or a master slave
>>> setup for JM HA specifically for Queryable access ( For flink jobs it is
>>> master slave )
>>> * Do we replicate state to other TMs for read optimization (
>>> specifically to avoid Hot Node issues ) ?
>>> * If the job goes down it seems the state is not accessible. What plans
>>> to we have to "separate concerns" for Queryable state.
>>>
>>> We consider Queryable State significant a feature Flink provides and
>>> would do the necessary leg work if there are certain gaps in it being
>>> trully considered a Highly Available Key Value store.
>>>
>>> Regards.
>>>
>>>
>>>
>>>
>>>
>>> On Mon, Mar 19, 2018 at 5:53 AM, Fabian Hueske <fh...@gmail.com>
>>> wrote:
>>>
>>>> Hi Vishal,
>>>>
>>>> In general, Queryable State should be ready to use.
>>>> There are a few things to consider though:
>>>>
>>>> - State queries are not synchronized with the application code, i.e.,
>>>> they can happen at the same time. Therefore, the Flink application should
>>>> not modify objects that have been put into or read from the state if you
>>>> are not using the RocksDBStatebackend (which creates copies by
>>>> deserialization).
>>>> - State will be rolled back after a failure. Hence, you can read writes
>>>> that are not "committed by a checkpoint".
>>>>
>>>> @Kostas, did I forget something?
>>>>
>>>> Best, Fabian
>>>>
>>>>
>>>>
>>>> 2018-03-18 16:50 GMT+01:00 Vishal Santoshi <vi...@gmail.com>:
>>>>
>>>>> To be more precise, is anything thing similar to
>>>>> https://engineering.linkedin.com/blog/2018/03/air-traffic
>>>>> -controller--member-first-notifications-at-linkedin . done in Samza,
>>>>> can be replicated with production level guarantees with Flink Queryable
>>>>> state ( as it stands currently version 1.5 ) ?
>>>>>
>>>>> On Fri, Mar 16, 2018 at 5:10 PM, Vishal Santoshi <
>>>>> vishal.santoshi@gmail.com> wrote:
>>>>>
>>>>>> We are making few decisions on use cases where Queryable state is a
>>>>>> natural fit https://ci.apache.org/projects
>>>>>> /flink/flink-docs-release-1.4/dev/stream/state/queryable_state.html
>>>>>>
>>>>>> Is Queryable state production ready ? We will go to 1.5 flnk if that
>>>>>> helps to make the case for the usage.
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
>
Re: Queryable State
Posted by Vishal Santoshi <vi...@gmail.com>.
Are there plans to address all or few of the above apart from the "JM LB
not possible" which seems understandable ?
On Mon, Mar 19, 2018 at 9:58 AM, Fabian Hueske <fh...@gmail.com> wrote:
> Queryable state is "just" an additional feature on regular keyed state.
> i.e., the only difference is that you can read the state from an outside
> application.
> Besides that it behaves exactly like regular application state
>
> Queryable state is (at the moment) designed to be accessible if a job runs.
> If the job fails (and recovers) or is manually taken down for maintenance,
> the state cannot be queried.
> It's not possible to put a load balancer in front of a JobManager. Only
> one JM is the active master that maintains a running job.
> State is also not replicated.
>
> Best, Fabian
>
>
> 2018-03-19 14:24 GMT+01:00 Vishal Santoshi <vi...@gmail.com>:
>
>> Those are understandable. I am more interested in a few things ( and may
>> be more that could be added )
>>
>> * As far as I can understand JM is the SPOF. Does HA become a necessity ?
>> * If there are 2 or more JM could we theoretically have a LB fronting
>> them ? Thus it is a peer to peer access ( Cassandra ) or a master slave
>> setup for JM HA specifically for Queryable access ( For flink jobs it is
>> master slave )
>> * Do we replicate state to other TMs for read optimization ( specifically
>> to avoid Hot Node issues ) ?
>> * If the job goes down it seems the state is not accessible. What plans
>> to we have to "separate concerns" for Queryable state.
>>
>> We consider Queryable State significant a feature Flink provides and
>> would do the necessary leg work if there are certain gaps in it being
>> trully considered a Highly Available Key Value store.
>>
>> Regards.
>>
>>
>>
>>
>>
>> On Mon, Mar 19, 2018 at 5:53 AM, Fabian Hueske <fh...@gmail.com> wrote:
>>
>>> Hi Vishal,
>>>
>>> In general, Queryable State should be ready to use.
>>> There are a few things to consider though:
>>>
>>> - State queries are not synchronized with the application code, i.e.,
>>> they can happen at the same time. Therefore, the Flink application should
>>> not modify objects that have been put into or read from the state if you
>>> are not using the RocksDBStatebackend (which creates copies by
>>> deserialization).
>>> - State will be rolled back after a failure. Hence, you can read writes
>>> that are not "committed by a checkpoint".
>>>
>>> @Kostas, did I forget something?
>>>
>>> Best, Fabian
>>>
>>>
>>>
>>> 2018-03-18 16:50 GMT+01:00 Vishal Santoshi <vi...@gmail.com>:
>>>
>>>> To be more precise, is anything thing similar to
>>>> https://engineering.linkedin.com/blog/2018/03/air-traffic
>>>> -controller--member-first-notifications-at-linkedin . done in Samza,
>>>> can be replicated with production level guarantees with Flink Queryable
>>>> state ( as it stands currently version 1.5 ) ?
>>>>
>>>> On Fri, Mar 16, 2018 at 5:10 PM, Vishal Santoshi <
>>>> vishal.santoshi@gmail.com> wrote:
>>>>
>>>>> We are making few decisions on use cases where Queryable state is a
>>>>> natural fit https://ci.apache.org/projects
>>>>> /flink/flink-docs-release-1.4/dev/stream/state/queryable_state.html
>>>>>
>>>>> Is Queryable state production ready ? We will go to 1.5 flnk if that
>>>>> helps to make the case for the usage.
>>>>>
>>>>
>>>>
>>>
>>
>
Re: Queryable State
Posted by Fabian Hueske <fh...@gmail.com>.
Queryable state is "just" an additional feature on regular keyed state.
i.e., the only difference is that you can read the state from an outside
application.
Besides that it behaves exactly like regular application state
Queryable state is (at the moment) designed to be accessible if a job runs.
If the job fails (and recovers) or is manually taken down for maintenance,
the state cannot be queried.
It's not possible to put a load balancer in front of a JobManager. Only one
JM is the active master that maintains a running job.
State is also not replicated.
Best, Fabian
2018-03-19 14:24 GMT+01:00 Vishal Santoshi <vi...@gmail.com>:
> Those are understandable. I am more interested in a few things ( and may
> be more that could be added )
>
> * As far as I can understand JM is the SPOF. Does HA become a necessity ?
> * If there are 2 or more JM could we theoretically have a LB fronting them
> ? Thus it is a peer to peer access ( Cassandra ) or a master slave setup
> for JM HA specifically for Queryable access ( For flink jobs it is master
> slave )
> * Do we replicate state to other TMs for read optimization ( specifically
> to avoid Hot Node issues ) ?
> * If the job goes down it seems the state is not accessible. What plans to
> we have to "separate concerns" for Queryable state.
>
> We consider Queryable State significant a feature Flink provides and would
> do the necessary leg work if there are certain gaps in it being trully
> considered a Highly Available Key Value store.
>
> Regards.
>
>
>
>
>
> On Mon, Mar 19, 2018 at 5:53 AM, Fabian Hueske <fh...@gmail.com> wrote:
>
>> Hi Vishal,
>>
>> In general, Queryable State should be ready to use.
>> There are a few things to consider though:
>>
>> - State queries are not synchronized with the application code, i.e.,
>> they can happen at the same time. Therefore, the Flink application should
>> not modify objects that have been put into or read from the state if you
>> are not using the RocksDBStatebackend (which creates copies by
>> deserialization).
>> - State will be rolled back after a failure. Hence, you can read writes
>> that are not "committed by a checkpoint".
>>
>> @Kostas, did I forget something?
>>
>> Best, Fabian
>>
>>
>>
>> 2018-03-18 16:50 GMT+01:00 Vishal Santoshi <vi...@gmail.com>:
>>
>>> To be more precise, is anything thing similar to
>>> https://engineering.linkedin.com/blog/2018/03/air-traffic
>>> -controller--member-first-notifications-at-linkedin . done in Samza,
>>> can be replicated with production level guarantees with Flink Queryable
>>> state ( as it stands currently version 1.5 ) ?
>>>
>>> On Fri, Mar 16, 2018 at 5:10 PM, Vishal Santoshi <
>>> vishal.santoshi@gmail.com> wrote:
>>>
>>>> We are making few decisions on use cases where Queryable state is a
>>>> natural fit https://ci.apache.org/projects
>>>> /flink/flink-docs-release-1.4/dev/stream/state/queryable_state.html
>>>>
>>>> Is Queryable state production ready ? We will go to 1.5 flnk if that
>>>> helps to make the case for the usage.
>>>>
>>>
>>>
>>
>
Re: Queryable State
Posted by Vishal Santoshi <vi...@gmail.com>.
Those are understandable. I am more interested in a few things ( and may be
more that could be added )
* As far as I can understand JM is the SPOF. Does HA become a necessity ?
* If there are 2 or more JM could we theoretically have a LB fronting them
? Thus it is a peer to peer access ( Cassandra ) or a master slave setup
for JM HA specifically for Queryable access ( For flink jobs it is master
slave )
* Do we replicate state to other TMs for read optimization ( specifically
to avoid Hot Node issues ) ?
* If the job goes down it seems the state is not accessible. What plans to
we have to "separate concerns" for Queryable state.
We consider Queryable State significant a feature Flink provides and would
do the necessary leg work if there are certain gaps in it being trully
considered a Highly Available Key Value store.
Regards.
On Mon, Mar 19, 2018 at 5:53 AM, Fabian Hueske <fh...@gmail.com> wrote:
> Hi Vishal,
>
> In general, Queryable State should be ready to use.
> There are a few things to consider though:
>
> - State queries are not synchronized with the application code, i.e., they
> can happen at the same time. Therefore, the Flink application should not
> modify objects that have been put into or read from the state if you are
> not using the RocksDBStatebackend (which creates copies by deserialization).
> - State will be rolled back after a failure. Hence, you can read writes
> that are not "committed by a checkpoint".
>
> @Kostas, did I forget something?
>
> Best, Fabian
>
>
>
> 2018-03-18 16:50 GMT+01:00 Vishal Santoshi <vi...@gmail.com>:
>
>> To be more precise, is anything thing similar to
>> https://engineering.linkedin.com/blog/2018/03/air-traffic
>> -controller--member-first-notifications-at-linkedin . done in Samza, can
>> be replicated with production level guarantees with Flink Queryable state (
>> as it stands currently version 1.5 ) ?
>>
>> On Fri, Mar 16, 2018 at 5:10 PM, Vishal Santoshi <
>> vishal.santoshi@gmail.com> wrote:
>>
>>> We are making few decisions on use cases where Queryable state is a
>>> natural fit https://ci.apache.org/projects/flink/flink-docs-release-1.4/
>>> dev/stream/state/queryable_state.html
>>>
>>> Is Queryable state production ready ? We will go to 1.5 flnk if that
>>> helps to make the case for the usage.
>>>
>>
>>
>
Re: Queryable State
Posted by Fabian Hueske <fh...@gmail.com>.
Hi Vishal,
In general, Queryable State should be ready to use.
There are a few things to consider though:
- State queries are not synchronized with the application code, i.e., they
can happen at the same time. Therefore, the Flink application should not
modify objects that have been put into or read from the state if you are
not using the RocksDBStatebackend (which creates copies by deserialization).
- State will be rolled back after a failure. Hence, you can read writes
that are not "committed by a checkpoint".
@Kostas, did I forget something?
Best, Fabian
2018-03-18 16:50 GMT+01:00 Vishal Santoshi <vi...@gmail.com>:
> To be more precise, is anything thing similar to https://engineering.
> linkedin.com/blog/2018/03/air-traffic-controller--member-
> first-notifications-at-linkedin . done in Samza, can be replicated with
> production level guarantees with Flink Queryable state ( as it stands
> currently version 1.5 ) ?
>
> On Fri, Mar 16, 2018 at 5:10 PM, Vishal Santoshi <
> vishal.santoshi@gmail.com> wrote:
>
>> We are making few decisions on use cases where Queryable state is a
>> natural fit https://ci.apache.org/projects/flink/flink-docs-release-1.4/
>> dev/stream/state/queryable_state.html
>>
>> Is Queryable state production ready ? We will go to 1.5 flnk if that
>> helps to make the case for the usage.
>>
>
>
Re: Queryable State
Posted by Vishal Santoshi <vi...@gmail.com>.
To be more precise, is anything thing similar to
https://engineering.linkedin.com/blog/2018/03/air-traffic-controller--member-first-notifications-at-linkedin
. done in Samza, can be replicated with production level guarantees with
Flink Queryable state ( as it stands currently version 1.5 ) ?
On Fri, Mar 16, 2018 at 5:10 PM, Vishal Santoshi <vi...@gmail.com>
wrote:
> We are making few decisions on use cases where Queryable state is a
> natural fit https://ci.apache.org/projects/flink/flink-docs-
> release-1.4/dev/stream/state/queryable_state.html
>
> Is Queryable state production ready ? We will go to 1.5 flnk if that helps
> to make the case for the usage.
>