You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by bastien dine <ba...@gmail.com> on 2019/11/15 21:56:46 UTC
Keyed raw state - example
Hello everyone,
I would like to know if anybody has a working example on how to declare a
keyed raw state ( in my case a keyedprocessoperator) and how to use it in
my UDF (keyedprocessfunction)?
Basicaly we have a huge problem with a ValueState w Rocksdb, getting
serialized for every element ( need to access it and update) so it's taking
a crazy amount of time and we would like to have it serialized only on
snapshot, so using Raw state is a possible good solution,
But i cannot find anyexample of it :/
Thanks and best regards,
Bastien DINE
Freelance
Data Architect / Software Engineer / Sysadmin
http://bastiendine.io
Re: Keyed raw state - example
Posted by bastien dine <ba...@gmail.com>.
Hello Congxian,
Thanks for your response,
Don't you have an example with an Operator extending the
AbstractUdfStreamOperator?
Using the context.getRawKeyedStateInputs() (& output to snapshots)
TimeService is reimplementing the whole stuff :/
------------------
Bastien DINE
Data Architect / Software Engineer / Sysadmin
bastiendine.io
Le lun. 18 nov. 2019 à 03:19, Congxian Qiu <qc...@gmail.com> a
écrit :
> Hi
> Currently, I think you can ref the implementation of timerservice[1]
> which used the raw keyed state, the snapshot happens in
> AbstractStreamOperator#snapshotState(), for using Raw State you need to
> implement a new operator[2]. There is an issue wants to give some example
> for raw state[2]
>
> [1]
> https://github.com/apache/flink/blob/master/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/InternalTimeServiceManager.java
> [2]
> https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/state/state.html#raw-and-managed-state
> [3] https://issues.apache.org/jira/browse/FLINK-14379
>
> Best,
> Congxian
>
>
> bastien dine <ba...@gmail.com> 于2019年11月16日周六 上午5:57写道：
>
>> Hello everyone,
>>
>> I would like to know if anybody has a working example on how to declare a
>> keyed raw state ( in my case a keyedprocessoperator) and how to use it in
>> my UDF (keyedprocessfunction)?
>>
>> Basicaly we have a huge problem with a ValueState w Rocksdb, getting
>> serialized for every element ( need to access it and update) so it's taking
>> a crazy amount of time and we would like to have it serialized only on
>> snapshot, so using Raw state is a possible good solution,
>> But i cannot find anyexample of it :/
>>
>> Thanks and best regards,
>>
>> Bastien DINE
>> Freelance
>> Data Architect / Software Engineer / Sysadmin
>> http://bastiendine.io
>>
>>
>>
>
Re: Keyed raw state - example
Posted by Congxian Qiu <qc...@gmail.com>.
Hi
Currently, I think you can ref the implementation of timerservice[1]
which used the raw keyed state, the snapshot happens in
AbstractStreamOperator#snapshotState(), for using Raw State you need to
implement a new operator[2]. There is an issue wants to give some example
for raw state[2]
[1]
https://github.com/apache/flink/blob/master/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/InternalTimeServiceManager.java
[2]
https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/state/state.html#raw-and-managed-state
[3] https://issues.apache.org/jira/browse/FLINK-14379
Best,
Congxian
bastien dine <ba...@gmail.com> 于2019年11月16日周六 上午5:57写道：
> Hello everyone,
>
> I would like to know if anybody has a working example on how to declare a
> keyed raw state ( in my case a keyedprocessoperator) and how to use it in
> my UDF (keyedprocessfunction)?
>
> Basicaly we have a huge problem with a ValueState w Rocksdb, getting
> serialized for every element ( need to access it and update) so it's taking
> a crazy amount of time and we would like to have it serialized only on
> snapshot, so using Raw state is a possible good solution,
> But i cannot find anyexample of it :/
>
> Thanks and best regards,
>
> Bastien DINE
> Freelance
> Data Architect / Software Engineer / Sysadmin
> http://bastiendine.io
>
>
>